Vehicular occupant detection arrangements

ABSTRACT

An arrangement in a vehicle for determining vehicle occupant position relative to a fixed structure within the vehicle including an array which receives an image of a portion of the passenger compartment in which the occupant is likely to be situated and a lens arranged between the array and the portion of the passenger compartment. The image may be changed by adjusting the lens, e.g., adjusting the focal length of the lens and/or the position of the lens relative to the array, by adjusting the array, e.g., the position of the array relative to the lens, and/or by using software to perform a focusing process. By changing the image to obtain the clearest image, i.e., focusing the image, a distance between the occupant and the fixed structure can be obtained based on the parameters of the change of the image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 09/476,255 filed Dec. 30, 1999, now U.S. Pat. No. 6,324,453,which in turn claims priority under 35 U.S.C. §1.119(e) of U.S.provisional patent application Ser. No. 60/114,507 filed Dec. 31, 1998.

This application is also a continuation-in-part of U.S. patentapplication Ser. No. 09/389,947 filed Sep. 3, 1999 which in turn is acontinuation-in-part of U.S. patent application Ser. No. 09/200,614,filed Nov. 30, 1998, now U.S. Pat. No. 6,141,432, which in turn is acontinuation of U.S. patent application Ser. No. 08/474,786 filed Jun.7, 1995, now U.S. Pat. No. 5,845,000, all of which are incorporated byreference herein.

This application is related to, but does not claim priority from U.S.patent application Ser. No. 07/878,571 filed May 5, 1992, now abandoned,U.S. patent application Ser. No. 08/040,978 filed Mar. 31, 1993, nowabandoned, U.S. patent application Ser. No. 08/505,036 filed Jul. 21,1995, now U.S. Pat. No. 5,653,462, U.S. patent application Ser. No.08/247,760 filed May 23, 1994, now abandoned, U.S. patent applicationSer. No. 08/640,068 filed Apr. 30, 1996, now U.S. Pat. No. 5,829,782,and U.S. patent application Ser. No. 08/239,978 filed May 9, 1994, nowabandoned, all of which incorporated by reference herein.

FIELD OF THE INVENTION

This invention relates to arrangements for detecting the presence, typeand position of occupants in vehicles and objects exterior of vehicles,e.g., in a driver's blind spot, primarily using optics.

BACKGROUND OF THE INVENTION

1. Prior Art on Out of position occupants and rear facing child seats

Whereas thousands of lives have been saved by airbags, a large number ofpeople have also been injured, some seriously, by the deploying airbag,and over 100 people have now been killed. Thus, significant improvementsneed to be made to airbag systems. As discussed in detail in U.S. Pat.No. 5,653,462 reference above, for a variety of reasons vehicleoccupants may be too close to the airbag before it deploys and can beseriously injured or killed as a result of the deployment thereof. Also,a child in a rear facing child seat that is placed on the right frontpassenger seat is in danger of being seriously injured if the passengerairbag deploys. For these reasons and, as firstly publicly disclosed inBreed, D. S. “How Airbags Work” presented at the InternationalConference on Seatbelts and Airbags in 1993, in Canada, occupantposition sensing and rear facing child seat detection systems arerequired.

Initially, these systems will solve the out-of-position occupant and therear facing child seat problems related to current airbag systems andprevent unneeded airbag deployments when a front seat is unoccupied.However, airbags are now under development to protect rear seatoccupants in vehicle crashes and all occupants in side impacts. A systemwill therefore be needed to detect the presence of occupants, determineif they are out-of-position and to identify the presence of a rearfacing child seat in the rear seat. Further automobiles are expected tohave eight or more airbags as protection is sought for rear seatoccupants and from side impacts. In addition to eliminating thedisturbance and possible harm of unnecessary airbag deployments, thecost of replacing these airbags will be excessive if they all deploy inan accident needlessly.

Inflators now exist which will adjust the amount of gas flowing to theairbag to account for the size and position of the occupant and for theseverity of the accident. The vehicle identification and monitoringsystem (VIMS) discussed in U.S. Pat. No. 5,829,782 will control suchinflators based on the presence and position of vehicle occupants or ofa rear facing child seat. As discussed more fully below, the instantinvention is an improvement on that VIMs system and uses an advancedoptical system comprising one or more CCD (charge coupled device) orCMOS arrays and particularly active pixel arrays plus a source ofillumination preferably combined with a trained neural network patternrecognition system.

Others have observed the need for an occupant out-of-position sensor andseveral methods have been disclosed in U.S. patents for determining theposition of an occupant of a motor vehicle. Each of these systems,however, has significant limitations. For example, in White et al. (U.S.Pat. No. 5,071,160), a single acoustic sensor and detector is describedand, as illustrated, is mounted lower than the steering wheel. White etal. correctly perceive that such a sensor could be defeated, and theairbag falsely deployed, by an occupant adjusting the control knobs onthe radio and thus they suggest the use of a plurality of such sensors.

Mattes et al. (U.S. Pat. No. 5,118,134) described a variety of methodsof measuring the change in position of an occupant including ultrasonic,active or passive infrared and microwave radar sensors, and an electriceye. The sensors measure the change in position of an occupant during acrash and use that information to access the severity of the crash andthereby decide whether or not to deploy the airbag. They are thus usingthe occupant motion as a crash sensor. No mention is made of determiningthe out-of-position status of the occupant or of any of the otherfeatures of occupant monitoring as disclosed in one or more of the abovecross-referenced patents and patent applications. It is interesting tonote that nowhere does Mattes et al. discuss how to use active orpassive infrared to determine the position of the occupant. As pointsout in one or more of the above cross-referenced patents and patentapplications, direct occupant position measurement based on passiveinfrared is probably not possible and, until very recently, was verydifficult and expensive with active infrared requiring the modulation ofan expensive GaAs infrared laser. Since there is no mentioned of theseproblems, the method of use contemplated by Mattes et al. must besimilar to the electric eye concept where position is measuredindirectly as they occupant passes by a plurality of longitudinallyspaced-apart sensors.

The object of an occupant out-of-position sensor is to determine thelocation of the head and/or chest of the vehicle occupant relative tothe airbag since it is the impact of either the head or chest with thedeploying airbag which can result in serious injuries. Both White et al.and Mattes et al. describe only lower mounting locations and theirsensors in front of the occupant such as on the dashboard or below thesteering wheel. Both such mounting locations are particularly prone todetection errors due to positioning of the occupant's hands, arms andlegs. This would require at least three, and preferably more, suchsensors and detectors and an appropriate logic circuitry which ignoresreadings from some sensors if such readings are inconsistent withothers, for the case, for example, where the driver's arms are theclosest objects to two of the sensors.

White et al. also describe the use of error correction circuitry,without defining or illustrating the circuitry, to differentiate betweenthe velocity of one of the occupant's hands as in the case where he/sheis adjusting the knob on the radio and the remainder of the occupant.Three ultrasonic sensors of the type disclosed by White et al. might, insome cases, accomplish this differentiation if two of them indicatedthat the occupant was not moving while the third was indicating that heor she was. Such a combination, however, would not differentiate betweenan occupant with both hands and arms in the path of the ultrasonictransmitter at such a location that they were blocking a substantialview of the occupant's head or chest. Since the sizes and drivingpositions of occupants are extremely varied, it is now believed thatpattern recognition systems and preferably trained pattern recognitionsystems, such as neural networks, are required when a clear view of theoccupant, unimpeded by his/her extremities, cannot be guaranteed.

Fujita et al., in U.S. Pat. No. 5,074,583, describe another method ofdetermining the position of the occupant but do not use this informationto suppress deployment if the occupant is out-of-position. In fact, thecloser the occupant gets to the airbag, the faster the inflation rate ofthe airbag is according to the Fujita et al. patent, which therebyincreases the possibility of injuring the occupant. Fujita et al. do notmeasure the occupant directly but instead determine his or her positionindirectly from measurement of the seat position and the vertical sizeof the occupant relative to the seat (occupant height). This occupantheight is determined using an ultrasonic displacement sensor mounteddirectly above the occupant's head.

As discussed above, the optical systems described herein are alsoapplicable for many other sensing applications both inside and outsideof the vehicle compartment such as for sensing crashes before they occuras described in patent application Ser. No. 08/239,978 cross-referencedabove, for a smart headlight adjustment system and for a blind spotmirror (also described in U.S. provisional patent application Ser. No.60/202,424).

2. Definitions

The use of pattern recognition is central to the instant invention aswell as to one or more of those disclosed in the cross-referencedpatents and patent applications above. “Pattern recongition” as usedherein will generally mean any system which processes a signal that isgenerated by an object, or is modified by interacting with an object, inorder to determine which one of a set of classes that the object belongsto. Such a system might determine only that the object is or is not amember of one specified class, or it might attempt to assign the objectto one of a larger set of specified classes, or find that it is not amember of any of the classes in the set. The signals processed aregenerally electrical signals coming from transducers which are sensitiveto either acoustic or electromagnetic radiation and, or electromagnetic,they can be either visible light, infrared, ultraviolet or radar or lowfrequency radiation as used in capacitive sensing systems.

A trainable or a trained pattern recognition system as used herein meansa pattern recognition system which is taught various patterns bysubjecting the system to a variety of examples. The most successful suchsystem is the neural network. Not all pattern recognition systems aretrained systems and not all trained systems are neural networks. Otherpattern recognition systems are based on fuzzy logic, sensor fusion,Kalman filters, correlation as well as linear and non-linear regression.Still other pattern recognition systems are hybrids of more than onesystem such as neural-fuzzy systems.

To “identify” as used herein will usually mean to determine that theobject belongs to a particular set or class. The class may be onecontaining, for example, all rear facing child seats, one containing allhuman occupants, or all human occupants not sitting in a rear facingchild seat depending on the purpose of the system. In the case where aparticular person is to be recognized, the set or class will containonly a single element, i.e., the person to be recognized.

To “ascertain the identity of” as used herein with reference to anobject will generally mean to determine the type or nature of the object(obtain information as to what the object is), i.e., that the object isan adult, an occupied rear facing child seat, an occupied front facingchild seat, an unoccupied rear facing child seat, an unoccupied frontfacing child seat, a child, a dog, a bag of groceries, etc.

An “occupying item” or “occupant” of a seat or “object” in a seat may bea living occupant such as a human being or a dog, another livingorganism such as a plant, or an inanimate object such as a box or bag ofgroceries.

In the description herein on anticipatory sensing, the term“approaching” when used in connection with the mention of an object orvehicle approaching another will usually mean the relative motion of theobject toward the vehicle having the anticipatory sensor system. Thus,in a side impact with a tree, the tree will be considered as approachingthe side of the vehicle and impacting the vehicle. In other words, thecoordinate system used in general will be a coordinate system residingin the target vehicle. The “target” vehicle is the vehicle that is beingimpacted. This convention permits a general description to cover all thecases such as where (i) a moving vehicle impacts into the side of astationary vehicle, (ii) where both vehicles are moving when theyimpact, or (iii) where a vehicle is moving sideways into a stationaryvehicle, tree or wall.

“Out-of-position” as used for an occupant will generally means that theoccupant, either the driver or a passenger, is sufficiently close to anoccupant protection apparatus (airbag) prior to deployment that he orshe is likely to be more seriously injured by the deployment eventitself than by the accident. It may also mean that the occupant is notpositioned appropriately in order to attain the beneficial, restrainingeffects of the deployment of the airbag. As for the occupant being tooclose to the airbag, this typically occurs when the occupant's head orchest is closer than some distance such as about 5 inches from thedeployment door of the airbag module. The actual distance value whereairbag deployment should be suppressed depends on the design of theairbag module and is typically farther for the passenger airbag than forthe driver airbag.

3. Patter recognition prior art

Japanese Patent No. 3-423337 (A) to Ueno discloses a device fordetecting the driving condition of a vehicle driver comprising a lightemitter for irradiating the face of the driver and a means for pickingup the image of the drive and storing it for later analysis. Means areprovided for locating the eyes of the driver and then the irises of theeyes and then determining if the driver is looking to the side orsleeping. Ueno determines the state of the eyes of the occupant ratherthan determining the location of the eyes relative to the other parts ofthe vehicle passenger compartment. Such a system can be defeated if thedriver is wearing glasses, particularly sunglasses, or another opticaldevice which obstructs a clear view of his/her eyes. Pattern recognitiontechnologies such as neural networks are not used.

U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rules toisolate the eyes and mouth of a driver and uses this information topermit the driver to control the radio, for example, or other systemswithin the vehicle by moving his eyes and/or mouth. Ando uses naturallight and illuminates only the head of the driver. He also makes no useof trainable pattern recognition systems such as neural networks, nor isthere any attempt to identify the contents of the vehicle nor of theirlocation relative to the vehicle passenger compartment. Rather, Ando islimited to control the vehicle devices by responding to motion of thedriver's mouth and eyes.

U.S. Pat. No. 5,298,732 to Chen also concentrates on locating the eyesof the driver so as to position a light filter between a light sourcesuch as the sun or the lights of an oncoming vehicle, and the driver'seyes. Chen does not explain in detail how the eyes are located but doessupply a calibration system whereby the driver can adjust the filter sothat it is at the proper position relative to his or her eyes. Chenreferences the use of automatic equipment for determining the locationof the eyes but does not describe how this equipment works. In anyevent, there is no mention of illumination of the occupant, monitoringthe position of the occupant, other that the eyes, determining theposition of the eyes relative to the passenger compartment, oridentifying any other object in the vehicle other than the driver'seyes. Also, there is no mention of the use of a trainable patternrecognition system.

U.S. Pat. No. 5,305,012 to Faris also describes a system for reducingthe glare from the headlights of an oncoming vehicle. Faris locates theeyes of the occupant utilizing two spaced apart infrared cameras usingpassive infrared radiation from the eyes of the driver. Again, Faris isonly interested in locating the driver's eyes relative to the sun oroncoming headlights and does not identify or monitor the occupant orlocate the occupant relative to the passenger compartment or the airbag.Also, Faris does not use trainable pattern recognition techniques suchas neural networks. Faris, in fact, does not even say how the eyes ofthe occupant are located but refers the reader to a book entitled RobotVision (1991) by Berthold Horn, published by MIT press, Cambridge, Mass.A review of this book did not appear to provide the answer to thisquestion. Also, Faris uses the passive infrared radiation rather thanilluminating the occupant with active infrared radiation or in generalelectromagnetic radiation.

The use of neural networks as the pattern recognition technology iscentral to several of the implementations of this invention since itmakes the monitoring system robust, reliable and practical. Theresulting algorithm created by the neural network program is usuallyonly a few dozen lines of code written in the C or C++ computer languageas opposed to typically hundreds of lines when the techniques of theabove patent to Ando, Chen and Faris are implemented. As a result, theresulting systems are easy to implement at a low cost, making thempractical for automotive applications. The cost of the CCD and CMOSarrays, for example, have been prohibitively expensive until recently,rendering their use for VIMS impractical. Similarly, the implementationof the techniques of the above referenced patents requires expensivemicroprocessors while the implementation with neural networks andsimilar trainable pattern recognition technologies permits the use oflow cost microprocessors typically costing less than $10 in largequantities.

The present invention preferably uses sophisticated trainable patternrecognition capabilities such as neural networks. Usually the data ispreprocessed, as discussed below, using various feature extractiontechniques. An example of such a pattern recognition system using neuralnetworks on sonar signals is discussed in two papers by Gorman R. P. andSejnowski, T. J. “Analysis of Hidden Units in a Layered Network Trainedto Classify Sonar Targets”, Neural Networks, Vol. 1. pp. 75-89, 1988,and “Learned Classification of Sonar Targets Using a Massively ParallelNetwork”, IEEE Transactions on Acoustics, Speech, and Signal Processing,Vol. 36, No. 7, July 1988. Examples of feature extraction techniques canbe found in U.S. Pat. No. 4,906,940 entitled “Process and Apparatus forthe Automatic Detection and Extraction of Features in Images andDisplays” to Green et al. Examples of other more advanced and efficientpattern recognition techniques can be found in U.S. Pat. No. 5,390,136entitled “Artificial Neuron and Method of Using Same and U.S. patentapplication Ser. No. 08/076,601 entitled “Neural Network and Method ofUsing Same” to S. T. Wang. Other examples include U.S. Pat. Nos.5,235,339 (Morrison et al.), 5,214,744 (Schweizer et al), 5,181,254(Schweizer et al), and 4,881,270 (Knecht et al). All of the abovereferences are included herein by reference.

4. Optics

Optics can be used in several configuration for monitoring the interiorof a passenger compartment of an automobile. In one known method, alaser optical system uses a GaAs infrared laser beam to momentarilyilluminate an object, occupant or child seat, in the manner as describedand illustrated in FIG. 8 of U.S. Pat. No. 5,829,782 cross-referencedabove. The receiver can be a charge-coupled device or CCD (a type of TVcamera), to receive the reflected light. The laser can either be used ina scanning mode, or, through the use of a lens, a cone of light can becreated which covers a large portion of the object. In theseconfigurations, the light can be accurately controlled to onlyilluminate particular positions of interest within the vehicle. In thescanning mode, the receiver need only comprise a single or a few activeelements while in the case of the cone of light, an array of activeelements is needed. The laser system has one additional significantadvantage in that the distance to the illuminated object can bedetermined as disclosed in the '462 patent.

In a simpler case, light generated by a non-coherent light emittingdiode device is used to illuminate the desired area. In this case, thearea covered is not as accurately controlled and a larger CCD or CMOSarray is required. Recently, however, the cost of CCD and CMOS arrayshas dropped substantially with the result that this configuration is nowthe most cost-effective system for monitoring the passenger compartmentas long as the distance from the transmitter to the objects is notneeded. If this distance is required, then the laser system, astereographic system, a focusing system, a combined ultrasonic and opticsystem, or a multiple CCD or CMOS array system as described herein isrequired.

A mechanical focusing system, such as used on some camera systems candetermine the initial position of an occupant but is too slow to monitorhis/her position during a crash. A distance measuring system based onfocusing is described in U.S. Pat. No. 5,193,124 (Subbarao) which caneither be used with a mechanical focusing system or with two cameras,the latter of which would be fast enough. Although the Subbarao patentprovides a good discussion of the camera focusing art and is thereforeincorporated herein by reference, it is a more complicated system thanis needed for the practicing of the instant invention. In fact, a neuralnetwork can also be trained to perform the distance determination basedon the two images taken with different camera settings or from twoadjacent CCD's and lens having different properties as the camerasdisclosed in Subbarao making this technique practical for the purposesof the instant invention. Distance can also be determined by the systemdisclosed in U.S. Pat. No. 5,003,166 (Girod) by the spreading ordefocusing of a pattern of structured light projected onto the object ofinterest. Distance can also be measured by using time of flightmeasurements of the electromagnetic waves or by multiple CCD or CMOSarrays as is a principle teaching of this invention.

In each of these cases, regardless of the distance measurement systemused, a trained pattern recognition system, as defined above, is used inthe instant invention to identify and classify, and in some cases tolocated, the illuminated object and its constituent parts.

5. Optics and acoustics

The laser systems described above are expensive due to the requirementthat they be modulated at a high frequency if the distance from theairbag to the occupant, for example, needs to be measured.

Both laser and non-laser optical systems in general are good atdetermining the location of objects within the two dimensional plane ofthe image and a pulsed laser radar system in the scanning mode candetermine the distance of each part of the image from the receiver bymeasuring the time of flight through range gating techniques. It is alsopossible to determine distance with the non-laser system by focusing asdiscussed above, or stereographically if two spaced apart receivers areused and, in some cases the mere location in the field of view can beused to estimate the position relative to the airbag, for example.Finally, a recently developed pulsed quantum well diode laser alsoprovides inexpensive distance measurements as discussed below.

Acoustic systems are additionally quite effective at distancemeasurements since the relatively low speed of sound permits simpleelectronic circuits to be designed and minimal microprocessor capabilityis required. If a coordinate system is used where the z axis is from thetransducers to the occupant, acoustics are good at measuring zdimensions while simple optical systems using a single CCD are good atmeasuring x and y dimensions. The combination of acoustics and optics,therefore, permits all three measurements to be made from one locationwith low cost components as discussed in commonly assigned U.S. Pat.Nos. 5,845,000 and 5,835,613 cross-referenced above.

One example of a system using these ideas is an optical system whichfloods the passenger seat with infrared light coupled with a lens andCCD or CMOS array which receives and displays the reflected light and ananalog to digital converter (ADC), or frame grabber, which digitizes theoutput of the CCD or CMOS and feeds it to an Artificial Neural Network(ANN) and other pattern recognition system for analysis. This systemuses an ultrasonic transmitter and receiver for measuring the distancesto the objects located in the passenger seat. The receiving transducerfeeds its data into an ADC and from there the converted data is directedinto the ANN. The same ANN can be used for both systems therebyproviding full three-dimension data for the ANN to analyze. This system,using low cost components, will permit accurate identification anddistance measurements not possible by either system acting alone. If aphased array system is added to the acoustic part of the system, theoptical part can determine the location of the driver's ears, forexample, and the phased array can direct a narrow beam to the locationand determine the distance to the occupant's ears.

Although the use of ultrasound for distance measurement has manyadvantages, it also has some drawbacks. First, the speed of sound limitsthe rate at which the position of the occupant can be updated toapproximately 10 milliseconds, which though sufficient for most cases,is marginal if the position of the occupant is to be tracked during avehicle crash. Second, ultrasound waves are diffracted by changes in airdensity that can occur when the heater or air conditioner is operated orwhen there is a high-speed flow of air past the transducer. Third, theresolution of ultrasound is limited by its wavelength and by thetransducers, which are high Q tuned devices. Typically, the resolutionof ultrasound is one the order of about 2 to 3 inches. Finally, thefields from ultrasonic transducers are difficult to control so thatreflections from unwanted objects or surfaces add noise to the data.

6. Applications

The applications for this technology are numerous as described in thecopending patents and patent applications listed above. They include (i)the monitoring of the occupant for safety purposes to prevent airbagdeployment induced injuries, (ii) the locating of the eyes of theoccupant (driver) to permit automatic adjustment of the rear viewmirror(s), (iii) the location of the seat to place the occupant's eyesat the proper position to eliminate the parallax in a heads-up displayin night vision systems (iv) the location of the ears of the occupantfor optimum adjustment of the entertainment system, (v) theidentification of the occupant for security reasons, (vi) thedetermination of obstructions in the path of a closing door or window,(vii) the determination of the position of the occupant's shoulder sothat the seat belt anchorage point can be adjusted for the bestprotection of the occupant, (viii) the determination of the position ofthe rear of the occupants head so that the headrest can be adjusted tominimize whiplash injuries in rear impacts, (ix) anticipatory crashsensing, (x) blind spot detection, (xi) smart headlight dimmers, (xii)sunlight and headlight glare reduction and many others. In fact, overforty products alone have been identified based on the ability toidentify and monitor objects and parts thereof in the passengercompartment of an automobile truck.

7. Other Prior Art

European Patent Application No. 98110617.2 (Publication No. 0 885 782A1), corresponding to U.S. patent application Ser. No. 08/872,836 filedJun. 11, 1997, describes a purportedly novel motor vehicle controlsystem including a pair of cameras which operatively produce first andsecond images of a passenger area. A distance processor determines thedistances that a plurality of features in the first and second imagesare from the cameras based on the amount that each feature is shiftedbetween the first and second images. An analyzer processes thedetermined distances and determines the size of an object on the seat.Additional analysis of the distance also may determine movement of theobject and the rate of movement. The distance information also can beused to recognize predefined patterns in the images and this identifyobjects. An air bag controller utilizes the determined objectcharacteristics in controlling deployment of the air bag.

A paper entitled “Sensing Automobile Occupant Position with OpticalTriangualtion” by W. Chappelle, Sensors, December 1995, describes theuse of optical triangulation techniques for determining the presence andposition of people or rear-facing infant seats in the passengercompartment of a vehicle in order to guarantee the safe deployment of anair bag. The paper describes a system called the “Takata Safety Shield ”which purportedly makes high-speed distance measurements from the pointof air bag deployment using a modulated infrared beam projected from anLED source. Two detectors are provided, each consisting of an imaginglens and a position-sensing detector.

A paper entitled “An Interior Compartment Protection System based onMotion Detection Using CMOS Imagers” by S. B. Park et al., 1998 IEEEInternational Conference on Intelligent Vehicles, describes apurportedly novel image processing system based on a CMOS image sensorinstalled at the car roof for interior compartment monitoring includingtheft prevention and object recognition. One disclosed camera system isbased on a CMOS image sensor and a near infrared (NIR) light emittingdiode (LED) array.

A paper entitled “A 256×256 CMOS Brightness Adaptive Imaging Array withColumn-Parallel Digital Output” by C. Sodini et al., 1988 IEEEInternational Conference on Intelligent Vehicles, describes a CMOS imagesensor for intelligent transportation system applications such asadaptive cruise control and traffic monitoring. Among the purposednovelties is the use of a technique for increasing the dynamic range ina CMOS imager by a factor of approximately 20, which technique is basedon a previously described technique for CCD imagers.

A paper entitled “Intelligent System for Video Monitoring of VehicleCockpit” by S. Boverie et al., SAE Technical Paper Series No. 980613,Feb. 23-26, 1998, describes the installation of an optical retina sensorin the vehicle and several uses of this sensor. Possible users are saidto include observation of the driver's face (eyelid movement) and thedriver's attitude to allow analysis of the driver's vigilance level andwarn him/her about critical situations and observation of the frontpassenger seat to allow the determination of the presence of somebody orsomething located on the seat and to value the volumetric occupancy ofthe passenger for the purpose of optimizing the operating conditions forair bags.

Ishikawa et al. (U.S. Pat. No. 4,625,329) describes an image analyzer(M5 in FIG. 1) for analyzing the position of driver inclusion aninfrared light source which illuminates the driver's face and an imagedetector which receives light from the driver's face, determines theposition of facial feature, e.g., the eyes in three dimensions, and thusdetermines the position of the driver in three dimensions. A patternrecognition process is used to determine the position of the facialfeatures and entails converting the pixels forming the image to eitherblack or white based on intensity and conducting an analysis based onthe white area in order to find the largest contiguous white area andthe center point thereof. Based on the location of the center point ofthe largest contiguous white area, the driver's height is derived and aheads up display is adjusted so information is within the driver's fieldof view. The pattern recognition process can be applied to detect theeyes, mouth, or nose of the driver based on the differentiation betweenthe white and black areas.

Ando (U.S. Pat. No. 5,008,946) describes a system which recognizes animage and specifically ascertains the position of the pupils and mouthof the occupant to enable movement of the pupils and mouth to controlelectrical devices installed in the automobile. The system includes acamera which takes a picture of the occupant and applies algorithmsbased on pattern recognition techniques to analyze the picture,converted into an electrical signal, to determine the position ofcertain portions of the image, namely the pupils and mouth.

Masamori (U.S. Pat. No. 5,227,784) describes a system which is based onradar, specifically it is a collision avoidance system aimed atdetecting vehicles which are at some distance from the vehicle.

Suzuki et al. (U.S. Pat. No. 50,26,153) describes a vehicle trackingcontrol for continuously detecting the distance and direction to apreceding vehicle irrespective of background dark/light distribution. Inthis system, every vehicle must have a light on its rear that emits aconstant or time varying signal and two photoelectric sensors that zeroin on the light emitted from the preceding vehicle are used and therebydetermine both the distance and angular position of the precedingvehicle.

Krumm (U.S. Pat. No. 5,983,147) describes a system for determining theoccupancy of a passenger compartment including a pair of cameras mountedso as to obtain binocular stereo images of the same location in thepassenger compartment. A representation of the output from the camerasis compared to stored representations of known occupants and occupancysituations to determine which stored representation the output from thecameras most closely approximates. The stored representations includethat of the presence or absence of a person or an infant seat in thefront passenger seat.

Farmer et al. (U.S. Pat. No. 6,005,958) describes a method and systemfor detecting the type and position of a vehicle occupant utilizing asingle camera unit. The single camera unit is positioned at the driveror passenger side A-pillar in order to generate data of the frontseating area of the vehicle. The type and position of the occupant isused to optimize the efficiency and safety in controlling deployment ofan occupant protection device such as an air bag.

OBJECTS AND SUMMARY OF THE INVENTION

Principle objects and advantages of the optical sensing system inaccordance with the invention are:

1. To recognize the presence of a human on a particular seat of a motorvehicle and to use this information to affect the operation of anothervehicle system such as the airbag, heating and air conditioning, orentertainment systems, among others.

2. To recognize the presence of a human on a particular seat of a motorvehicle and then to determine his/her position and to use this positioninformation to affect the operation of another vehicle system.

3. To determine the position, velocity or size of an occupant in a motorvehicle and to utilize this information to control the rage of gasgeneration, or the amount of gas generated by an airbag inflator system.

4. To determine the presence or position of rear seated occupants in thevehicle and to use this information to affect the operation of a rearseat protection airbag for frontal, side and/or rear impacts.

5. To recognize the presence of a rear facing child seat on a particularseat of a motor vehicle and to use this information to affect theoperation of another vehicle system such as the airbag system.

6. To determine the approximate location of the eyes of a driver and touse that information to control the position of one or more of the rearview mirrors of the vehicle.

7. To monitor the position of the head of the vehicle driver anddetermine whether the driver is falling asleep or otherwise impaired andlikely to lose control of the vehicle and to use that information toaffect another vehicle system.

8. To provide an occupant position sensor which reliably permits, and ina timely manner, a determination to be made that the occupant isout-of-position, or will become out-of-position, and likely to beinjured by a deploying airbag and to then output a signal to suppressthe deployment of the airbag.

9. To provide an anticipatory sensor that permits accurateidentification of the about-to-impact object in the presence of snowand/or fog whereby the sensor is located within the vehicle.

10. To provide a smart headlight dimmer system which sense theheadlights from an oncoming vehicle or the tail lights of a vehicle infront of the subject vehicle and identifies these lights differentiatingthem from reflections from signs or the road surface and then sends asignal to dim the headlights.

11. To provide a blind spot detector which detects and categories anobject in the driver's blind spot or other location in the vicinity ofthe vehicle, and warns the driver in the event the driver begins tochange lanes, for example, or continuously informs the driver of thestate of occupancy of the blind spot.

12. To provide a occupant position determination in a sufficiently shorttime that the position of an occupant can be tracked during a vehiclecrash.

13. To provide an occupant vehicle interior monitoring system which isnot affected by temperature or thermal gradients.

14. To provide an occupant vehicle interior monitoring system which hashigh resolution to improve system accuracy and permits the location ofbody parts of the occupant to be determined.

15. To provide an occupant vehicle interior monitoring system whichreduces the glare from sunlight and headlights by imposing a filterbetween the eyes of an occupant and the light source.

16. To provide a camera system for interior and exterior monitoring,which can adjust on a pixel by pixel basis for the intensity of thereceived light.

These and other objects and advantages will become apparent from thefollowing description of the preferred embodiments of the vehicleidentification and monitoring system of this invention.

Briefly though, in order to achieve at least one of the objects, avehicle including an arrangement for determining vehicle occupantposition relative to a fixed structure within the vehicle comprises anarray structured and arranged to receive an image of a portion of thepassenger compartment of the vehicle in which the occupant is likely tobe situated, a lens arranged between the array and the portion of thepassenger compartment, adjustment means for changing the image receivedby the array, and processor means coupled to the array and theadjustment means. The processor means determine, upon changing by theadjustment means of the image received by the array, when the image isclearest whereby a distance between the occupant and the fixed structureis obtainable based on the determination by the processor means when theimage is clearest. The image may be changed by adjusting the lens, e.g.,adjusting the focal length of the lens and/or the position of the lensrelative to the array, by adjusting the array, e.g., the position of thearray relative to the lens, and/or by using software to perform afocusing process.

The array may be arranged in several advantageous locations on thevehicle, e.g., on an A-pillar of the vehicle, above a top surface of aninstrument panel of the vehicle and on an instrument panel of thevehicle and oriented to receive an image reflected by a windshield ofthe vehicle. The array may be a CCD array with an optical liquid crystalor electrochromic glass filter coupled to the array for filtering theimage of the portion of the passenger compartment. The array could alsobe a CMOS array.

In a preferred embodiment, the processor means are coupled to anoccupant protection device and control the occupant protection devicebased on the distance between the occupant and the fixed structure. Forexample, the occupant protection device could be an airbag wherebydeployment of the airbag is controlled by the processor means. Theprocessor means may be any type of data processing unit such as amicroprocessor.

This arrangement could be adapted for determining distance between thevehicle and exterior objects, in particular, objects in a blind spot ofthe driver. In this case, such an arrangement would comprise an arraystructure and arranged to receive an image of an exterior environmentsurrounding the vehicle containing at least one object, a lens arrangedbetween the array and the exterior environment, adjustment means forchanging the image received by the array, and processor means coupled tothe array and the adjustment means. The processor means determine, uponchanging by the adjustment means of the image received by the array,when the image is clearest whereby a distance between the object and thevehicle is obtainable based on the determination by the processor meanswhen the image is clearest. As before, the image may be changed byadjusting the lens, e.g., adjusting the focal length of the lens and/orthe position of the lens relative to the array, by adjusting the array,e.g., the position of the array relative to the lens, and/or by usingsoftware to perform a focusing process. The array may be a CCD arraywith an optional liquid crystal or electrochromic glass filter coupledto the array for filtering the image of the portion of the passengercompartment. The array could also be a CMOS array. In a preferredembodiment, the processor means are coupled to an occupant protectiondevice and control the occupant protection device based on the distancebetween the occupant and the fixed structure. For example, the occupantprotection device could be an airbag whereby deployment of the airbag iscontrolled by the processor means. The processor means may be any typeof data processing unit such as a microprocessor.

Also, at least one of the above-listed objects is achieved by anarrangement for determining vehicle occupant presence, type and/orposition relative to a fixed structure within the vehicle, the vehiclehaving a front seat and an A-pillar. The arrangement comprises a firstarray mounted on the A-pillar of the vehicle and arranged to receive animage of a portion of the passenger compartment in which the occupant islikely to be situated, and processor means coupled to the first arrayfor determining the presence, type and/or position of the vehicleoccupant based on the image of the portion of the passenger compartmentreceived by the first array. The processor means preferably are arrangedto utilize a pattern recognition technique, e.g., a trained neuralnetwork, sensor fusion, fuzzy logic. The processor means can determinethe vehicle occupant presence, type and/or position based on the imageof the portion of the passenger compartment received by the first array.

In some embodiments, a second array is arranged to receive an image ofat least a part of the same portion of the passenger compartment as thefirst array. The processor means are coupled to the second array anddetermine the vehicle occupant presence, type and/or position based onthe images of the portion of the passenger compartment received by thefirst and second arrays. The second array may be arranged at a centralportion of a headliner of the vehicle between sides of the vehicle.

The determination of the occupant presence, type and/or position can beused in conjunction with a reactive component, system or subsystem sothat the processor means control the reactive component, system orsubsystem based on the determination of the occupant presence, typeand/or position. For example, if the reactive component system orsubsystem is an airbag assembly including at least one airbag, theprocessor means control one or more deployment parameters of theairbag(s).

The arrays may be CCD arrays with an optional liquid crystal orelectrochromic glass filter coupled to the array for filtering the imageof the portion of the passenger compartment. The arrays could also beCMOS arrays, active pixel cameras and HDRC cameras.

Another embodiment of the invention is an arrangement for obtaininginformation about a vehicle occupant within the vehicle which comprisestransmission means for transmitting a structured pattern of light, e.g.,polarized light, into a portion of the passenger compartment in whichthe occupant is likely to be situated, an array arranged to receive animage of the portion of the passenger compartment, and processor meanscoupled to the array for analyzing the image of the portion of thepassenger compartment to obtain information about the occupant. Thetransmission means and array are proximate one another and theinformation obtained about the occupant is a distance from the locationof the transmission means and the array. The processor means obtain theinformation about the occupant utilizing a pattern recognitiontechnique. The information about the occupant can be used in conjunctionwith a reactive component, system or subsystem so that the processormeans control the reactive component, system or subsystem based on thedetermination of the occupant presence, type and/or position. Forexample, if the reactive component, system or subsystem is an airbagassembly including at least one airbag, the processor means control oneor more deployment parameters of the airbag(s).

BRIEF DESCRIPTION OF THE DRAWINGS

The following drawings are illustrative of embodiments of the inventionand are not meant to limit the scope of the invention as encompassed bythe claims.

FIG. 1A is a side planar view, with certain portions removed or cutaway, of a portion of the passenger compartment of a vehicle showingseveral preferred mounting locations of interior vehicle monitoringsensors shown particularly for sensing the vehicle driver illustratingthe wave pattern from a CCD or CMOS optical position sensor mountedalong the side of the driver or centered above his or her head.

FIG. 1B is a view as in FIG. 1A illustrating the wave pattern from anoptical system using an infrared light source and a CCD or CMOS arrayreceiver using the windshield as a reflection surface and showingschematically the interface between the vehicle interior monitoringsystem of this invention and an instrument panel mounted inattentivenesswarning light or buzzer and reset button.

FIG. 1C is a view as in FIG. 1A illustrating the wave pattern from anoptical system using an infrared light source and a CCD or CMOS arrayreceiver where the CCD or CMOS array receiver is covered by a lenspermitting a wide angle view of the contents of the passengercompartment.

FIG. 1D is a view as in FIG. 1A illustrating the wave pattern from apair of small CCD or CMOS array receivers and one infrared transmitterwhere the spacing of the CCD or CMOS arrays permits an accuratemeasurement of the distance to features on the occupant.

FIG. 1E is a view as in FIG. 1A illustrating the wave pattern from a setof ultrasonic transmitter/receivers where the spacing of the transducersand the phase of the signal permits an accurate focusing of theultrasonic beam and thus the accurate measurement of a particular pointon the surface of the driver.

FIG. 2A is a side view, with certain portions removed or cut away, of aportion of the passenger compartment of a vehicle showing preferredmounting locations of optical interior vehicle monitoring sensors.

FIG. 2B is a perspective view, with certain portions removed or cutaway, of a portion of the passenger compartment of a vehicle showingsome preferred mounting locations of optical interior vehicle monitoringsensors.

FIG. 3 is a circuit schematic illustrating the use of the vehicleinterior monitoring sensor used as an occupant position sensor inconjunction with the remainder of the inflatable restraint system.

FIG. 4 is a schematic illustrating the circuit of an occupantposition-sensing device using a modulated infrared signal, beatfrequency and phase detector system.

FIG. 5 is a side planer view with parts cutaway and removed of a vehicleshowing the passenger compartment containing a driver and a preferredmounting location for an optical occupant position sensor for use inside impacts and also of an optical rear of occupant's head locator foruse with a headrest adjustment system to reduce whiplash injuries inrear impact crashes.

FIG. 6 is a side plan view of the interior of an automobile, withportions cut away and removed, with two optical occupant heightmeasuring sensors, one mounted into the headliner above the occupant'shead and the other mounted onto the A-pillar and also showing a seatbeltassociated with the seat where the seatbelt has an adjustable upperanchorage point which is automatically adjusted corresponding to theheight of the occupant.

FIG. 7 is a perspective view of a vehicle about to impact the side ofanother vehicle showing the location of the various parts of theanticipatory sensor system of this invention.

FIG. 7A is an enlarged view of the section designated 7A in FIG. 7.

FIG. 8 is a side planar view, with certain portions removed or cut away,of a portion of the passenger compartment illustrating a sensor forsensing the headlights of an oncoming vehicle and/or the taillights of aleading vehicle used in conjunction with an automatic headlight dimmingsystem.

FIG. 9 is a side planar view with parts cutaway and removed of a subjectvehicle and an oncoming vehicle, showing the headlights of the oncomingvehicle and the passenger compartment of the subject vehicle, containingdetectors of the driver's eyes and detectors for the headlights of theoncoming vehicle and the selective filtering of the light of theapproaching vehicle's headlights through the use of a liquid crystalfilter in the windshield.

FIG. 9A is an enlarged view of the section designated 9A in FIG. 9.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring now to the drawings wherein the same reference numerals referto the same or similar elements, a section of the passenger compartmentof an automobile is shown generally as 100 in FIGS. 1A-1D. A driver 101of a vehicle sits on a seat 102 behind a steering wheel 103, whichcontain an airbag assembly 104. Airbag assembly 104 may be integratedinto the steering wheel assembly or coupled to the steering wheel 103.Five transmitter and/or receiver assemblies 110, 111, 112, 113, and 114are positioned at various places in the passenger compartment (thespecific locations of which are set forth below) to determine thelocation of various parts of the driver, e.g., the head, chest andtorso, relative to the airbag and to otherwise monitor the interior ofthe passenger compartment. Monitoring of the interior of the passengercompartment can entail detecting the presence or absence of the driverand passengers, differentiating between animate and inanimate objects,detecting the presence of occupied or unoccupied child seats,rear-facing or forward-facing, and identifying and ascertaining theidentity of the occupying items in the passenger compartment. Processormeans such as control circuitry 120 is connected to thetransmitter/receiver assemblies 110-114 and controls the transmissionfrom the transmitters, if a transmission component is present in theassemblies, and captures the return signals from the receivers, if areceiver component is present in the assemblies. Control circuitry 120usually contains analog to digital converters (ADCs) or a frame grabber,a microprocessor containing sufficient memory and appropriate softwareincluding pattern recognition algorithms, and other appropriate drivers,signal conditioners, signal generators, etc. Usually, in any givenimplementation, only three or four of the transmitter/receiverassemblies would be used depending on their mounting locations asdescribed below.

With respect to the connection between the transmitter/receiverassemblies 110-114 and the control circuitry 120, a portion of thisconnection is shown as wires. It should be understood that all of theconnections between the transmitter/receiver assemblies 110-114 and thecontrol circuitry 120 may be wires, either individual wires leading fromthe control circuitry 120 to each of the transmitter/receiver assemblies110-114 or one or more wire buses.

With respect to the position of the control circuitry 120 in thedashboard of the vehicle, this position is for illustration purposesonly and does not limit the location of the control circuitry 120.Rather, the control circuitry 120 may be located anywhere convenient ordesired in the vehicle.

It is contemplated that a system and method in accordance with theinvention can include a single transmitter and multiple receivers, eachat a different location. Thus, each receiver would not be associatedwith a transmitter forming transmitter/receiver assemblies. Rather, forexample, with reference to FIG. 1A, only element 110 would constitute atransmitter/receiver assembly and elements 111, 112, 113, 114 would bereceivers only.

On the other hand, it is conceivable that in some implementations, asystem and method in accordance with the invention include a singlereceiver and multiple transmitters. Thus, each transmitter would not beassociated with a receiver forming transmitter/receiver assemblies.Rather, for example, with reference to FIG. 1A, only element 110 wouldconstitute a transmitter/receiver assembly and elements 111, 112, 113,114 would be transmitters only.

FIG. 1A illustrates a typical wave pattern of transmitted waves fromtransmitter/receiver assembly 111, which is mounted on the side of thevehicle passenger compartment above the front, driver's side door.Transmitter/receiver assembly 114, shown overlaid ontotransmitter/receiver 111, is actually mounted in the center headliner ofthe passenger compartment (and thus between the driver's seat and thefront passenger seat), near the dome light, and is aimed toward thedriver. Typically there will be symmetrical installation for thepassenger side of the vehicle. That is, a transmitter/receiver assemblywould be arranged above the front, passenger side door and anothertransmitter/receiver assembly would be arranged in the center headliner,near the dome light, and aimed toward the front, passenger side door.

In a preferred embodiment, each transmitter/receiver assembly 111,114comprises an optical transducer that will generally be used inconjunction with another optical transmitter/receiver assembly such asshown at 110, 112 and 113, which act in a similar manner. These opticaltransmitter/receiver assemblies are comprised of an optical transmitter,which may be an infrared LED (or possibly a near infrared (NIR)LED), alaser with a diverging lens or a scanning laser assembly, and a receiversuch as a CCD or CMOS array and particularly an active pixel CMOS cameraor array or a HDRL or HDRC camera or array as discussed below. Thetransducer assemblies map the location of the occupant(s), objects andfeatures thereof, in a two or three-dimensional image as will now bedescribed in more detail.

An active pixel camera is a special camera which has the ability toadjust the sensitivity of each pixel of the camera similar to the mannerin which an iris adjusts the sensitivity of a camera. Thus, the activepixel camera automatically adjusts to the incident light on apixel-by-pixel basis. An active pixel camera differs from an activeinfrared sensor in that an active infrared sensor, such as the typeenvisioned by Mattes et al., is generally a single pixel sensor thatmeasures the reflection of infrared light from an object.

A dynamic pixel camera is a camera having a plurality of pixels andwhich provides the ability to pick and choose which pixels should beobserved, as long as they are contiguous.

An HDRC camera is a type of active pixel camera where the dynamic rangeof each pixel is considerably broader. An active pixel cameramanufactured by the Photobit Corporation has a dynamic range of 70 dbwhile an IMS Chips camera, an HDRC camera manufactured by anothermanufacturer, has a dynamic range of 120 db. Thus, the HDRC camera has a100,000 times greater range of light sensitivity than the Photobitcamera.

In a preferred implementation, four transducer assemblies are positionedaround the seat to be monitored, each comprising an LED with a diverginglens and a CMOS array. Although illustrated together, the illustratingsource in many cases will not be co-located with the receiving array.The LED emits a controlled angle, 120° for example, diverging cone ofinfrared radiation that illuminates the occupant from both sides andfrom the front and rear. This angle is not to be confused with the filedangle used in ultrasonic systems. With ultrasound, extreme care isrequired to control the field of the ultrasonic waves so that they willnot create multipath effects and add noise to the system. With infrared,there is no reason, in the implementation now being described, otherthan to make the most efficient use of the infrared energy, why theentire vehicle cannot be flooded with infrared energy either from manysmall sources or from a few bright ones.

The image from each array is used to capture two dimensions of occupantposition information, thus, the array of assembly 110 positioned on theA-pillar, which is approximately 25% of the way laterally across theheadliner in front of the driver, provides a both vertical andtransverse information on the location of the driver. A similar viewfrom the rear is obtained from the array of assembly 113 positionedbehind the driver on the roof of the vehicle and above the seatbackpotion of the seat 102. As such, assembly 113 also provides bothvertical and transverse information on the location of the driver.Finally, arrays of assemblies 111 and 114 provide both vertical andlongitudinal driver location information. Another preferred location isthe headliner centered directly above the seat of interest. The positionof the assemblies 110-114 may differ from that shown in the drawings. Inthe invention, in order that the information from two or more of theassemblies 110-114 may provide a three-dimensional image of theoccupant, or portion of the passenger compartment, the assemblies shouldnot be arranged side-by-side. A side-by-side arrangement as used inseveral prior art references discussed above, will provide twoessentially identical views with the difference being a lateral shift.This does not enable a three-dimensional view of the occupant.

If each receiving array of assemblies 110, 111, 113, 114 contains amatrix of 100 by 100 pixels, then 40,000 (4×100×100) pixels or dataelements of information will be created each time the systeminterrogates the driver seat, for example. There are many pixels of eachimage that can be eliminated as containing not useful information. Thistypically includes the corner pixels, back of the seat and other areaswhere an occupant cannot reside. This pixel pruning can typically reducethe number of pixels by up to 50 percent resulting in approximately20,000 remaining pixels. The output from each array is then comparedwith a series of stored arrays representing different unoccupiedpositions of the seat, seatback, steering wheel etc. For each array,each of the stored arrays is subtracted from the acquired array and theresults analyzed to determine which subtraction resulted in the bestmatch. The best match is determined by such things as the total numberof pixels reduced below the threshold level, or the minimum number ofremaining detached pixels, etc. Once this operation is completed for allfour images, the position of the movable elements within the passengercompartment has been determined. This includes the steering wheel angle,telescoping position, seatback angle, headrest position, and seatposition. This information can be used elsewhere by other vehiclesystems to eliminate sensors that are currently being used to sense suchpositions of these complements. Alternately, the sensors that arecurrently on the vehicle for sensing these complement positions can beused to simplify processes described above.

Each receiving array may also be a 256×256 CMOS pixel array as describedin the paper by C. Sodini et al. referenced above.

Al alternate technique of differentiating between the occupant and thevehicle is to use motion. If the images of the passenger seat arecompared over time, reflections from fixed objects will remain staticwhereas reflections from vehicle occupants will move. This movement canbe used to differentiate the occupant from the background.

Following the subtraction process described above, each image nowconsists of typically as many as 50 percent fewer pixels leaving a totalof approximately 10,000 pixels remaining. The resolution of the imagesin each array can now be reduced by combining adjacent pixels andaveraging the pixel values. This results in a reduction to a total pixelcount of approximately 1000. The matrices of information that containsthe pixel values is now normalized to place the information in alocation in the matrix which is independent of the seat position. Theresulting normalized matrix of 1000 pixel values is now used as inputinto an artificial neural network and represents the occupancy of theseat independent of the position of the occupant.

The neural network has been previously trained on a significant numberof occupants of the passenger compartment. The number of such occupantsdepends strongly on whether the driver or the passenger seat is beinganalyzed. The variety of seating states or occupancies of the passengerseat is vastly greater than that of the driver seat. For the driverseat, a typical training set will consist of approximately 100 differentvehicle occupancies. For the passenger seat, this number can exceed1000. These numbers are used for illustration purposes only and willdiffer significantly from vehicle model to vehicle model.

The neural network is now used to determine which of the storedoccupancies most closely corresponds to the measured data. The output ofthe neural network is an index of the setup that was used duringtraining that most closely matches the current measured state. Thisindex is used to locate stored information from the matched trainedoccupancy. Information that has been stored for the trained occupancytypically includes the locus of the centers of the chest and head of thedriver, as well as the approximate radius of pixels which is associatedwith this center to define the head area, for example. For the case ofFIG. 1A, it is now known from this exercise where the head, chest, andperhaps the eyes and ears, of the driver are most likely to be locatedand also which pixels should be tracked in order to know the preciseposition of the driver's head and chest. What has been described aboveis the identification process.

The normalization process conducted above created a displacement valuefor each of the CCD or CMOS arrays in the four assemblies 110, 111, 113,114 which can now be used in reverse to find the precise location of thedriver's head and chest or chest, for example, relative to the knownlocation of the airbag. From the vehicle geometry, and the head andchest location information, a choice can now be made as to whether totrack the head or chest for dynamic out-of-position.

Tracking of the motion of the occupant's head or chest can be done usinga variety of techniques. One preferred technic is to use differentialmotion, that is, by subtracting the current image from the previousimage to determine which pixels have changed in value and by looking atthe leading edge of the changed pixels and the width of the changedpixel field, a measurement of the movement of the pixels of interest,and thus the driver, can be readily accomplished. Alternately, acorrelation function can be derived which correlates the pixels in theknown initial position of the head, for example, with pixels that werederived from the latest image. The displacement of the center of thecorrelation pixels would represent the motion of the head of theoccupant. Naturally, a wide variety of other techniques will be nowobvious to those skilled in the art.

These are many mathematical techniques that can be applied to simplifythe above process. One technique used in military pattern recognition,for example, uses the Fourier transform of particular areas in an imageto match with known Foureir transforms of known images. In this manner,the identification and location can be determined simultaneously. Thereis even a technique used for target identification whereby the Fouriertransforms are compared optically. Other techniques utilize thresholdingto limit the pixels that will be analyzed by any of these processes.Other techniques search for particular features and extract thosefeatures and concentrate merely on the location of certain of thesefeatures. (See for example the Kage et al. artificial retina publicationreferenced above which, together with the references cited therein, isincluded herein by reference).

The principle used in this preferred implementation of the invention isto use images of different views of the occupant to correlate with knownimages that were used to train a neural network for vehicle occupancy.Then, carefully measured positions of the known images are used tolocate particular parts of the occupant such as his or her head, chest,eyes, ears, mouth, etc. An alternate approach is to make athree-dimensional map of the occupant and to precisely locate thesefeatures using neural networks, fuzzy logic or other rules. One methodof obtaining a three-dimensional map is to utilize a scanning laserradar system where the laser is operated in a pulse mode and thedistance from the object being illuminated is determined using rangegating in a manner similar to that described in various patents on micropower impulse radar to Mean. (See for example U.S. Pat. Nos. 5,457,394and 5,521,600).

The scanning portion of the pulse laser radar device can be accomplishedusing rotating mirrors, mechanical motors, or preferably, a solid statesystem, for example, one utilizing TeO₂ as an optical diffractioncrystal with lithium niobate crystals driven by ultrasound (althoughother solid state systems not necessarily using TeO₂ and lithium niobatecrystals could also be used). An alternate method is to use amicromachined mirror, which is supported at its center and caused todeflect by miniature coils. Such a device has been used to providetwo-dimensional scanning to a laser. This has the advantage over theTeO₂ - lithium niobate technology in that it is inherently smaller andlower cost and provides two-dimensional scanning capability in one smalldevice. The maximum angular deflection that can be achieved with thisprocess is on the order of about 10 degrees. Thus, a diverging lens willbe needed for the scanning system.

An alternate method of obtaining three-dimensional information from ascanning laser system is to use multiple arrays to replace the singlearrays used in FIG. 1A. In the case, the arrays, are displaced from eachother and, through triangulation, the location of the reflection fromthe illumination by a laser beam of a point on the object can bedetermined in a manner that is understood by those skilled in the art.

One important point concerns the location and number of opticalassemblies. It is possible to use fewer than four such assemblies with aresulting loss in accuracy. The number of four was chosen so that eithera forward or rear assembly or either side assemblies can be blocked by anewspaper, for example, without seriously degrading the performance ofthe system. Since drivers rarely are reading newspapers while driving,fewer than four arrays are usually adequate for the driver side.

The particular locations of the optical assemblies were chosen to givethe most accurate information as to the locations of the occupant. Thisis based on an understanding of what information can be best obtainedfrom a visual image. There is a natural tendency on the part of humansto try to gauge distance from the optical sensors directly. This, as canbe seen above, is at best complicated involving focusing systems,stereographic systems, multiple arrays and triangulation, time of flightmeasurement, etc. What is not intuitive to humans is to not try toobtain this distance directly from apparatus or techniques associatedwith the mounting location. Whereas ultrasound is quite good formeasuring distances from the transducer (the z-axis), optical systemsare better at measuring distances in the vertical and lateral directions(the x and y-axes). Since the precise locations of the opticaltransducers are known, that is, the geometry of the transducer locationsis known relative to the vehicle, there is no need to try to determinethe displacement of an object of interest from the transducer (thez-axis) directly. This can more easily done indirectly by anothertransducer. That is, the z-axis to one transducer is the x-axis toanother.

Ultrasonic transducers are relatively good at measuring the distancealong a radius to a reflective object. An optical array, such asdisclosed herein, on the other hand, can get accurate measurements intwo dimensions, the lateral and vertical dimensions relative to thetransducer. If we assume that the optical array has dimensions of 100 by100 as compared to an ultrasonic sensor that has a single dimension of100, an optical array can give therefore 100 times as much informationas the ultrasonic array. Most importantly, this vastly greater amount ofinformation does not cost significantly more to obtain then theinformation from the ultrasonic sensor.

As illustrated in FIGS. 1A-1D, the optical sensors are typically locatedat the positions where the desired information is available with thegreatest resolution. These positions are typically in the center frontand center rear of the occupancy seat and at the center on each side andtop. This is in contrast to the optimum location for ultrasonic sensors,which are the corners of such a rectangle that outlines the seatedvolume.

Systems based on ultrasonic and neural networks have been verysuccessful in analyzing the seated state of both the passenger anddriver seats of automobiles. Such systems are now going into productionfor preventing airbag deployment when a rear facing child seat or andout-of-position occupant is present. The ultrasonic systems, however,suffer from certain natural limitations that prevent the system accuracyfrom getting better than about 99 percent. These limitations relate tothe fact that the wavelength of ultrasound is typically between 3 and 8mm. As a result, unexpected results occur which are due partially to theinterference of reflections from different surfaces. Additionally,commercially available ultrasonic transducers are tuned devices thatrequire several cycles before they transmit significant energy andsimilarly require several cycles before they effectively receive thereflected signals. This requirement has the effect of smearing theresolution of the ultrasound to the point that, for example, using aconventional 40 kHz transducer, the resolution of the system isapproximately three inches.

In contrast, the wavelength of infrared is less than one micron and nosignificant interferences occur. Similarly, the system is not tuned andtherefore is theoretically sensitive to a very few cycles. As a result,resolution of the optical system is determined by the pixel spacing inthe CCD or CMOS arrays. For this application, typical arrays have beenchosen to be 100 pixels by 100 pixels and therefore the space beingimaged can be broken up into pieces that are significantly less than 1cm in size. Naturally, if greater resolution is required arrays havinglarger numbers of pixels are readily available. Another advantage ofoptical systems is that special lenses can be used to magnify thoseareas where the information is most critical and operate at reducedresolution where this is not the case. For example, the area closest tothe at-risk zone in front of the airbag can be magnified. This is notpossible with ultrasonic systems.

To summarize, although ultrasonic neural network systems are operatingwith high accuracy, they do not totally eliminate the problem of deathsand injuries caused by airbag deployments. Optical systems, on the otherhand, at little increase in cost, have the capability of virtually 100percent accuracy. Additional problems of ultrasonic systems arise fromthe slow speed of sound and diffraction caused by variations is airdensity. The slow sound speed limits the rate at which data can becollected and thus eliminates the possibility of tracking the motion ofan occupant during a high speed crash.

In the case of FIG. 1A, transmitter/receiver assemblies 110-114 emitinfrared waves that reflect off of the head and chest of the driver andreturn thereto. Periodically, the device, as commanded by controlcircuitry 120, transmits a pulse of infrared waves and the reflectedsignal is detected by the same or a different device. The transmitterscan either transmit simultaneously or sequentially. An associatedelectronic circuit and algorithm in control circuitry 120 processes thereturned signals as discussed above and determines the location of theoccupant in the passenger compartment. This information is then sent tothe crash sensor and diagnostic circuitry, which may also be resident incontrol circuitry 120 (programmed within a control module), whichdetermines if the occupant is close enough to the airbag that adeployment might, but itself, cause injury which exceeds that whichmight be caused by the accident itself. In such a case, the circuitdisables the airbag system and thereby prevents its deployment. In analternate case, the sensor algorithm assesses the probability that acrash requiring an airbag is in process and waits until that probabilityexceeds an amount that is dependent on the position of the occupant.Thus, for example, the sensor might decide to deploy the airbag based ona need probability assessment of 50%, if the decision must be madeimmediately for an occupant approaching the airbag, but might wait untilthe probability rises to 95% for a more distant occupant. In thealternative, the crash sensor and diagnostic circuitry optionallyresident in control circuitry 120 may tailor the parameters of thedeployment (time to initiation of deployment, rate of inflation, rate ofdeflation, deployment time, etc.) based on the current position andpossibly velocity of the occupant, e.g., a depowered deployment.

Although a driver system has been illustrated, the front and rear seatpassenger systems would be similar.

In another implementation, the sensor algorithm may determine the ratethat gas is generated to affect the rate that the airbag is inflated. Inall of these cases, the position of the occupant is used to affect thedeployment of the airbag as to whether or not it should be deployed atall, the time of deployment and/or the rate of inflation.

The algorithm in control circuitry 120 can also be designed to determinethe location of the occupant's eyes either directly or indirectlythrough a determination of the location of the occupant and anestimation of the position of the eyes therefrom. As such, the positionof the rear view mirror 105 can be adjusted to optimize the driver'usethereof.

Weight sensors 130 are also included in the system shown in FIG. 1A.Although strain gage type sensors are schematically illustrated mountedto the supporting structure of the seat 102, any other type of weightsensor can be used. Strain gage weight sensors are described in detailin co-pending U.S. patent application Ser. No. 09/193,209 that isincluded herein by reference as if it were entirely incorporated herein.Weight can be used to confirm the occupancy of the seat, i.e., thepresence or absence of an occupant as well as whether the seat isoccupied by a light or heavy object. In the latter case, a measuredweight of less than 60 pounds is often determinative of the presence ofa child seat whereas a measured weight of greater than 60 pounds isoften indicative of the absence of a child seat. The weight sensors 130can also be sued to determine the weight distribution of the occupant ofthe seat and thereby ascertain whether the occupant is moving and theposition of the occupant. As such, the weight sensors 130 could be usedto confirm the position of the occupant. The measured weight ordistribution thereof can also be used in combination with the data fromthe transmitter/receiver assemblies 110-114 to provide an identificationof the occupants in the seat.

The accuracy of the optical occupant sensor is critically dependent uponthe accuracy of the camera. The dynamic range of light within a vehicleexceeds 120 decibels. When a car is driving at night, for example, verylittle light is available whereas when driving in a bright sunlight,especially in a convertible, the light intensity can overwhelm mostcameras. Additionally, the camera must be able to adjust rapidly tochanges and light caused by, for example, the emergence of the vehiclefrom tunnel, or passing by other obstructions such as trees, buildings,other vehicles, etc. which temporarily block the sun and cause astrobing effect at frequencies approaching 1 kHz.

Recently, improvements have been made to CMOS cameras that havesignificantly increased their dynamic range. New logarithmic highdynamic range technology such as developed by IMS Chips of Stuttgart,Germany, is now available in HDRC (High Dynamic Range CMOS) cameras.This technology provides a 120 dB dynamic intensity response at eachpixel in a mono chromatic mode. The technology has a 1 million to onedynamic range at each pixel. This prevents blooming, saturation andflaring normally associated with CMOS and CCD camera technology. Thissolves a problem that will be encountered in an automobile when goingfrom a dark tunnel into bright sunlight. Such a range would even exceedthe 120 dB intensity.

There is also significant infrared radiation from bright sunlight andfrom incandescent lights within the vehicle. Such situations may evenexceed the dynamic range of the HDRC camera and additional filtering maybe required. Changing the bias on the receiver array, the use of amechanical iris, or of electrochromic glass or liquid crystal canprovide this filtering on a global basis but not at a pixel level.Filtering can also be used with CCD arrays, but the amount of filteringrequired is substantially greater than for the HDRC camera.

Liquid crystals operate rapidly and give as much as a dynamic range of10,000 to 1 but may create a pixel interference affect. Electrochromicglass operates more slowly but more uniformly thereby eliminating thepixel affect. The pixel effect arises whenever there is one pixel devicein front of another. This results in various aliasing, Moire patternsand other ambiguities. One way of avoiding this is to blur the image.Another solution is to use a large number of pixels and combine groupsof pixels to form one pixel of information and thereby to blur the edgesto eliminate some of the problems with aliasing and Moire patterns.

One straightforward approach is the use of mechanical iris. Standardcameras already have response times of several tens of millisecondsrange. They will switch, for example, in a few frames on a typical videocamera (1 frame=0.033 seconds). This is sufficiently fast forcategorization but much too slow for dynamic out-of-position tracking.

An important feature of IMS Chips HDRC camera is that the full dynamicrange is available at each pixel. Thus, if there are significantvariations in the intensity of light within the vehicle, and therebyfrom pixel to pixel, such as would happen when sunlight streams andthrough a window, the camera can automatically adjust and provide theoptimum exposure on a pixel by pixel basis. The use of the camera havingthis characteristic is very beneficial to the invention described hereinand contributes significantly to system accuracy. CCDs have a ratherlimited dynamic range due to their inherent linear response andconsequently cannot come close to matching the performance of humaneyes. A key advantage of the IMS Chips HDRC camera is its logarithmicresponse which comes closest to matching that of the human eye.

Another approach, which is applicable in some vehicles, is to record animage without the infrared illumination and then a second image with theinfrared illumination and to then subtract the first image from thesecond image. In this manner, illumination caused by natural sourcessuch as sunlight or even from light bulbs within the vehicle can besubtracted out. Naturally, using the logarithmic pixel system of the IMSChips camera care must be taken to include the logarithmic effect duringthe subtraction process. For some cases, natural illumination such asfrom the sun, light bulbs within the vehicle, or radiation emitted bythe object itself can be used alone without the addition of a specialsource of infrared illumination.

Other imaging systems such as CCD arrays can also of course be used withthis invention. However, the techniques will be quite different sincethe camera is very likely to saturate when bright light is present andto require the full resolution capability when the light is dim.Generally, when practicing this invention the interior of the passengercompartment will be illuminated with infrared radiation.

There are other bright sources of infrared that must be accounted for.These include the sun and any light bulbs that may be present inside thevehicle. This lack of a high dynamic range inherent with the CCDtechnology requires the use of an iris, liquid crystal, orelectrochromic glass filter to be placed between the camera and thescene. Even with these filters however, some saturation will take placewith CCD cameras under bright sun or incandescent lamp exposure. Thissaturation reduces the accuracy of the image and therefore the accuracyof the system. In particular the training regimen that must be practicedwith CCD cameras is more severe since all of the saturation cases mustbe considered since the camera is unable to appropriately adjust. Thus,although CCD cameras can be use, HDRC logarithmic cameras such asmanufactured by IMS Chips are preferred. They not only provide asignificantly more accurate image but also significantly reduce theamount of training effort and associated data collection that must beundertaken during the development of the neural network algorithm orother computational intelligence system. In some applications, it ispossible to use other more deterministic image processing or patternrecognition systems than neural networks.

Another very important feature of the HDRC camera from IMS Chips is thatthe shutter time is constant at less than 100 ns irrespective ofbrightness of the scene. The pixel data arrives at constant ratesynchronous with the internal imager clock. Random access to each pixelfacilitates high=speed intelligent access to any sub=frame (block) sizeor sub-sampling ratio and a trade-off of frame speed and frame sizetherefore results. For example, a scene with 128K pixels per frame canbe taken at 120 frames per second, or about 8 milliseconds per frame,whereas a sub-frame can be taken in run at as high as 4000 frames persecond with 4K pixels per frame. This combination allows the maximumresolution for the identification and classification part of theoccupant sensor problem while permitting a concentration on thoseparticular pixels which track the head or chest, as described above, fordynamic out-of-position tracking. In fact the random access features ofthese cameras can be used to track multiple parts of the imagesimultaneously while ignoring the majority of the image, and do so atvery high speed. For example, the head can be tracked simultaneouslywith the chest by defining two separate sub-frames that need not beconnected. This random access pixel capability, therefore, is optimallysuited or recognizing the tracking vehicle occupants. It is also suitedfor monitoring the environment outside of the vehicle for purposes ofblind spot detection, collision avoidance and anticipatory sensing.Photobit Corporation of 135 North Los Robles Ave., Suite 700, Pasadena,Calif. 91101 manufactures another camera with some characteristicssimilar to the IMS Chips camera. Other competitive cameras can beexpected to appear on the market.

Photobit refers to their Active Pixel Technology as APS. According toPhotobit, in the APS, both the photodetector and readout amplifier arepart of each pixel. This allows the integrated charge to be convertedinto a voltage in the pixel that can then be read out over X-Y wiresinstead of using a charge domain shift register as in CCDs. This columnand row addressability (similar to common DRAM) allows for window ofinterest readout (windowing) which can be utilized for on chipelectronic pan/tilt and zoom. Windowing provides added flexibility inapplications, such as disclosed herein, needing image compression,motion detection or target tracking. The APS utilizes intra-pixelamplification in conjunction with both temporal and fixed pattern noisesuppression circuitry (i.e. correlated double sampling), which producesexceptional imagery in terms of wide dynamic range (˜75 dB) and lownoise (˜15 e- rms noise floor) with low fixed pattern noise (<0.15%sat). Unlike CCDs, the APS is not prone to column streaking due toblooming pixels. This is because CCDs rely on charge domain shiftregisters that can leak charge to adjacent pixels when the CCD registersoverflows. Thus, bright lights “bloom” and cause unwanted streaks in theimage. The active pixel can drive column busses at much greater ratesthan passive pixel sensors and CCDs. On-chip analog-to-digitalconversion (ADC) facilitates driving high speed signals off chip. Inaddition, digital output is less sensitive to pickup and crosstalk,facilitating computer and digital controller interfacing whileincreasing system robustness. A high speed APS recently developed for acustom binary output application produced over 8,000 frames per second,at a resolution of 128×128 pixels. It is possible to extend this designto a 1024×1024 array size and achieve greater than 1000 frames persecond for machine vision. All of these features are important to manyapplications of this invention.

These advanced cameras, as represented by the HDRC and the APS cameras,now make it possible to more accurately monitor the environment in thevicinity of the vehicle. Heretofore, the large dynamic range ofenvironmental light has either blinded the cameras when exposed tobright light or else made them unable to record images when the lightlevel was low. Even the HDRC camera with its 120 dB dynamic range may bemarginally sufficient the handle the fluctuations in environmental lightthat occur. Thus, the addition of a electrochromic, liquid crystal, orother similar filter may be necessary. This is particularly true forcameras such as the Photobit APS camera with its 75 dynamic range.

At about 120 frames per second, these cameras are adequate for caseswhere the relative velocity between vehicles is low. There are manycases, however, where this is not the case and a much higher monitoringrate is required. This occurs for example, in collision avoidance andanticipatory sensor applications. The HDRC camera is optimally suitedfor handling these cases since the number of pixels that are beingmonitored can be controlled resulting in a frame rate as high as about4000 frames per second with a smaller number of pixels.

Another key advantage of the HDRC camera is that it is quite sensitiveto infrared radiation in the 0.8 to 1 manometer wavelength range. Thisrange is generally beyond visual range for humans permitting this camerato be used with illumination sources that are not visible to the humaneye. Naturally, a notch filter is frequently used wit the camera toeliminate unwanted wavelengths. These cameras are available from theInstitute for Microelectronics (IMS Chips), Allamndring 30a, D-70579Stuttgart, Germany with a variety of resolutions ranging from 512 by 256to 720 by 576 pixels and can be custom fabricated for the resolution andresponse time required.

An optical infrared transmitter and receiver assembly is shown generallyat 112 in FIG. 1B and is mounted onto the instrument panel facing thewindshield. Assembly 112 can either be recessed below the upper face ofthe instrument panel or mounted onto the upper face of the instrumentpanel. Assembly 112, shown enlarged, comprises a source of infraredradiation, or another form of electromagnetic radiation, and a CCD orCMOS array of typically 160 pixels by 160 pixels. In this embodiment,the windshield is used to reflect the illumination light provided by theinfrared radiation toward the objects in the passenger compartment andalso reflect the light being reflected back by the objects in thepassenger compartment, in a manner similar to the “heads-up” displaywhich is now being offered on several automobile models. The “heads-up”display, of course, is currently used only to display information to thedriver and is not used to reflect light from the driver to a receiver.Once again, unless one of the distance measuring systems as describedbelow is used, this system alone cannot be used to determine distancesfrom the objects to the sensor. Its main purpose is objectidentification and monitoring. Depending on the application, separatesystems can be sued of the driver and for the passenger. In some cases,the cameras located in the instrument panel which receive lightreflected off of the windshield can be co-located with multiple lenseswhereby the respective lenses aimed at the driver and passenger seatsrespectively.

Assembly 112 is actually about two centimeters in diameter and is showngreatly enlarged in FIG. 1B. Also, the reflection area on the windshieldis considerably smaller than illustrated and special provisions are madeto assure that this area of the windshield is flat and reflective as isdone generally when heads-up displays are used. For cases where there issome curvature in the windshield, it can be at least partiallycompensated for by the CCD optics.

When using the surface of the windshield as a reflector in infraredradiation, care must be taken to assure that the desired reflectivity atthe frequency of interest is achieved. Mirror materials, such as metalsand other special materials manufactured by Eastman Kodak, have areflectivity for infrared frequencies that is substantially higher thanat visible frequencies. They are thus candidates for coatings to beplaced on the windshield services of this purpose. If two spaced apartCCD arrays are used, then the distance to the various objects within thepassenger compartment can be found by suing a triangulation algorithmwhich locates similar features on both images and determines theirrelative location on the images. An alternate method is to use a lenswith a short focal length. In this case, the lens is mechanicallyfocused, e.g., automatically, directly or indirectly, by the controlcircuitry 120, to determine the clearest image and thereby obtain thedistance to the object. This is similar to certain camera auto-focusingsystems such as one manufactured by Fuji of Japan. Naturally, othermethods can be used as described in the patents and patent applicationsreferenced above.

Instead of focusing the lens, the lens could be moved relative to thearray to thereby adjust the image on the array. Instead of moving thelens, the array could be moved to achieve the proper force. In addition,it is also conceivable that software could be used to focus the imagewithout moving the lens or the array.

Once a vehicle interior monitoring system employing a sophisticatedpattern recognition system, such as a neural network, is in place, it ispossible to monitor the motions of the driver over time, and his/herresponse to various stimuli, and determine if he or she is fallingasleep, has becomes incapacitated or otherwise unable to operate thevehicle. IN such an event, the vehicle can be caused to respond in anumber of different ways. One such system is illustrated in FIG. 1B andconsists of a monitoring system having the traducer assembly 112 coupledto a microprocessor in control circuitry 120 which is programmed tocompare the motions of the driver over time and trained to recognizechanges in behavior representative of becoming incapacitated, e.g., theeyes blinking erratically and remaining closed for ever longer periodsof time. If the system determines that there is a reasonable probabilitythat the driver has fallen asleep, for example, then it can activate analarm, e.g., turn on a warning light shown here as 124 or send a warningsound. If the driver fails to respond to the warning by pushing a button122, for example, then the horn and lights of the vehicle can beoperated in a manner to warn other vehicles and the vehicle may bebrought to a stop. Naturally, other responses can also be programmed andother tests of driver attentiveness can be used without resorting toattempting to monitor the motions of the driver's eyes.

The use of the windshield as a reflector is particularly useful whenmonitoring the eyes of the driver. The reflections from the cornea arehighly directional as every driver knows whose lights have reflected offthe eyes of an animal on the roadway. For this to be effective, the eyesof the driver must be looking at the radiation source. Since the driveris presumably looking through the windshield, the source of theradiation must also come from the windshield and the reflections fromthe driver's eyes must also be in the direction of the windshield. Usingthis technique, the time that the driver spends looking through thewindshield can be monitored and if that time drops below some thresholdvalue it can be presumed that the driver is not attentive and may besleeping or otherwise incapacitated.

An even more sophisticated system of monitoring the behavior of thedriver is to track the driver's eye motions using such techniques as aredescribed in: Friedman et al., U.S. Pat. No. 4,648,052 entitled “EyeTracker Communication System”; Heyner et al., U.S. Pat. No. 4,720,189entitled “Eye Position Sensor”; Hutchinson, U.S. Pat. No. 4,836,670entitled “Eye Movement Detector”; and Hutchinson, U.S. Pat. No.4,950,069 entitled “Eye Movement Detector With Improved Calibration andSpeed”, all of which are included herein by reference as well as U.S.Pat. Nos. 5,008,946 and 5,305,012 referenced above. The detection of theimpaired driver in particular can be best determined by thesetechniques. These systems make use of pattern recognition techniquesplus, in many cases, the transmitter and CCD receivers must beappropriately located so that the reflection off of the cornea of thedriver's eyes can be detected as discussed in the above referencedpatents. The size of the CCD arrays used herein permits their location,sometimes in conjunction with a reflective windshield, where thiscorneal reflection can be detected with some difficulty. Sunglasses orother items can interfere with this process.

The location of the eyes of the driver, for this application, is greatlyfacilitated by the teachings of this invention as described above.Although others have suggested the use of eye motions and cornealreflections for drowsiness determination, up until now there has notbeen a practical method for locating the driver's eyes with sufficientprecision and reliability as to render this technique practical. Also,although sunglasses might defeat such a system, most drowsiness causedaccidents happen at night where it is less likely that sunglasses areworn.

The eye tracker systems discussed above are facilitated by the instantinvention since one of the main purposes of determining the location ofthe driver's eyes either by directly locating them with trained patternrecognition technology or by inferring their location from the locationof the driver's head, is so that the seat can be automaticallypositioned to place the driver's eyes into the “eye-ellipse”. theeye-ellipse is the proper location for the driver's eyes to permitoptimal operation of the vehicle and for the location of the mirrorsetc. Thus, if the location of the driver's eyes are known, then thedriver can be positioned so that his or her eyes are precisely situatedin the eye ellipse and the reflection off of the eye can be monitoredwith a small eye tracker system. Also, by ascertaining the location ofthe driver's eyes, a rear view mirror positioning device can becontrolled to adjust the mirror 105 to an optimal position.

In addition to finding the location of the eyes, the location of theears is becoming more important. Many automobile accidents are now beingcaused by driver's holding on and talking into cellular phones. Vehiclenoise significantly deteriorates the quality of the sound heard by thedriver from speakers. This problem can be solved through the use ofhypersound and by knowing the location of the ears of the driver.Hypersound permits the precise focusing of sound wavers along a linefrom the speaker with little divergence of the sound field. Thus, if thelocations of the ears of the driver are known, the sound can beprojected to them directly thereby overcoming much of the vehicle noise.In addition to the use of hypersound, directional microphones are wellknown in the microphone art which are very sensitive to sound comingfrom a particular direction. If the driver has been positioned so thathis eyes are in the eye ellipse, then the location of the driver's mouthis also accurately known and a fixed position directional microphone canbe sued to selectively sense sound emanating from the mouth of thedriver. In many cases, the sensitivity of the microphone can be designedto include a large enough are such that most motions of the driver'headcan be tolerated. Alternatively the direction of the microphone can beadjusted using motors or the like. Systems of noise cancellation nowalso become possible if the ear locations are precisely known and noisecanceling microphones as described in copending U.S. provisional patentapplication Ser. No. 60/110,973, which is included herein by reference,if the location of the driver's mouth is known.

Infrared waves are shown coming from the front and back transducerassemblies 110 and 113 in FIG. 1C.

FIG. 1D illustrates two optical systems each having a source of infraredradiation and a CCD or CMOS array receiver. The price of such arrays hasdropped dramatically recently making them practical for interior andexterior vehicle monitoring. In this embodiment, transducers 110 and 113are CMOS arrays having 160 pixels by 160 pixels covered by a lens. Insome applications, this can create a “fisheye” effect whereby light froma wide variety of directions can be captured. One such transducer placedby the dome light or other central position in the vehicle headliner,such as the transducer designated 113, can monitor the entire vehicleinterior with sufficient resolution to determine the occupancy of thevehicle, for example. CCD's such as those used herein are available fromMarshall Electronics Inc. of Culver City, Calif. A fisheye lens is “. .. a wide-angle photographic lens that covers an angle of about 180°,producing a circular image with exaggerated foreshortening in the centerand increasing distortion toward the periphery”. (The American HeritageDictionary of the English Language. Third Edition, 1992 by HoughtonMifflin Company). This distortion of a fisheye lens can be substantiallychanged by modifying the shape of the lens to permit particular portionsof the interior passenger compartment to be observed. Also, in manycases the full 180° is not desirable and a lens which captures a smallerangle may be used. Although primarily spherical lenses are illustratedherein, it is understood that the particular lens design will depend onthe location in the vehicle and the purpose of the particular receiver.

CCD arrays are in common use in television cameras, for example, toconvert an image into an electrical signal. For the purposes herein, aCCD will be defined to include all devices, including CMOS arrays, APSarrays, artificial retinas and particularly HDRC arrays, which arecapable of converting light frequencies, including infrared, visible andultraviolet, into electrical singles. The particular CCD array used formany of the applications disclosed herein is implemented on a singlechip that is less than two centimeters on a side. Data from the CCDarray is digitized and sent serially to an electronic circuit (at timesdesignated 120 herein) containing a microprocessor for analysis of thedigitized data. In order to minimize the amount of data that needs to bestored, initial processing of the image data takes place as it is beingreceived from the CCD array, as discussed in more detail above. In somecases, some image processing can take place on the chip such asdescribed in the Kage et all artificial retina article referenced above.

One method of determining distance to an object directly withoutresorting to range finders, which require multiple arrays, is to use amechanical focusing system. However, the use of such an apparatus iscumbersome, expensive, and slow and has questionable reliability. Analternative is to use the focusing systems described in the abovereferenced U.S. Pat. Nos. 5,193,124 and 5,003,166, however, such systemsrequire expensive hardware and/or elaborate algorithms. Anotheralternative is illustrated in FIG. 1D where transducer 116 is aninfrared source having a wide transmission angle such that the entirecontents of the front driver's seat is illuminated. Receiving CCDtransducers 117 and 118 are shown spaced apart so that a sterographicanalysis can be made by the control circuitry 120. This circuitry 120contains a microprocessor with appropriate pattern recognitionalgorithms along with other circuitry as described above. In this case,the desired feature to be located is first selected from one of the tworeturned images from either CCD transducer 117 or 118. The software thendetermines the location of the same feature, through correlationanalysis or other methods, on the other image and thereby, throughanalysis familiar to those skilled in the art, determines the distanceof the feature from the transducers.

Transducers 116-118 are illustrated mounted onto the A-pillar of thevehicle, however, since these transducers are quite small, typicallyapproximately 2 cm on a side, they could alternately be mounted onto thewindshield itself, or other convenient location which provides a clearview of the portion of the passenger compartment being monitored. Otherpreferred mounting locations include the headliner above and also theside of the seat.

With respect to the connection between the transducers 110-114 and116-118 and the control circuitry 120, a portion of this connection isshown as wires. It should be understood that all of the connectionsbetween the transducers 110-114 and 116-118 and the control circuitry120 may be wires, either individual wires leading from the controlcircuitry 120 to each of the transducers 110-114 and 116-118 or one ormore wire buses.

With respect to the position of the control circuitry 120 in thedashboard of the vehicle, this position is for illustration purposesonly and does not limit the location of the control circuitry 120.Rather, the control circuitry 120 may be located anywhere convenient ordesired in the vehicle.

A new class of laser range finders has particular application here. Thisproduct, as manufactured by Power Spectra, Inc. of Sunnyvale, Calif., isa GaAs pulsed laser device which can measure up to 30 meters with anaccuracy of <2 cm and a resolution of <1 cm. This system is implementedin combination with transducer 116 and one of the receiving transducers117 or 118 may thereby be eliminated. Once a particular feature of anoccupying item of the passenger compartment has been located, thisdevice is used in conjunction with an appropriate aiming mechanism todirect the laser beam to that particular feature. The distance to thatfeature is then known to within 2 cm and with calibration even moreaccurately. In addition to measurements within the passengercompartment, this device has particular applicability in anticipatorysensing and blind spot monitoring applications exterior to the vehicle.An alternate technology using range gating to measure the time of flightof electromagnetic pulses with even between resolution can be developedbased on the teaching of the McEwan patents listed above and includedherein by reference.

A more accurate acoustic system for determining the distance to aparticular object, or a part thereof, in the passenger compartment isexemplified by transducers 111A in FIG. 1E. In this case, threeultrasonic transmitter/receivers are shown spaced apart mounted onto theA-pillar of the vehicle. The A-pillar is the forward most roof supportpillar and also supports the windshield. Due to the wavelength, it isdifficult to get a narrow beam using ultrasonics without either usinghigh frequencies that have limited range or a large transducer. Acommonly available 40 kHz transducer, for example, is about 1 cm. indiameter and emits a sonic wave that spreads at about a sixty-degreeangle. To reduce this angle requires making the transducer larger indiameter. An alternate solution is to use several transducers and tophase the transmissions so that they arrive at the intended part of thetarget in phase. Reflections form the selected part of the target arethen reinforced whereas reflections from adjacent parts encounterinterference with the result that the distance to the brightest portionwithin the vicinity of interest can be determined. By varying the phaseof transmission from the three transducers 111A, the location of areflection source on a curved line can be determined. In order to locatethe reflection source in space, at least one additionaltransmitter/receiver is required which is not co-linear with the others.The accuracy of the measurement can be determined by those skilled inthe art of phased array radar as the relevant equations are applicablehere. The waves shown in FIG. 1E coming from the three transducers 111Aare actually only the portions of the waves which arrive at the desiredpoint in space together in phase. The effective direction of these wavestreams can be varied by changing the transmission phase between thethree transmitters. A determination of the approximate location of apoint of interest on the occupant is accomplished by the CCD array andappropriate analysis and the phasing of the ultrasonic transmitters isdetermined so that the distance to the desired point can be determined.

FIG. 2A is a side view, with certain portions removed or cut away, of aportion of the passenger compartment of a vehicle showing preferredmounting locations of optical interior vehicle monitoring sensors(transmitter/receiver assemblies or transducers) 110, 111A, 113, 114,210, 211A, 213, 214, and 224. Each of these sensors is illustrated ashaving a lens and is shown enlarged in size for clarity. In a typicalactual device, the diameter of the lens is approximately 2 cm and itprotrudes from the mounting surface by approximately 1 cm. This smallsize renders these devices almost unnoticeable by vehicle occupants.Since these sensors are optical, it is important that the lens surfaceremains relatively clean. Control circuitry 120, which is coupled toeach transducer, contains a self-diagnostic feature where the imagereturned by a transducer is compared with a stored image and theexistence of certain key features is verified. If a receiver fails thistest, a warning is displayed to the driver which indicates that cleaningof the lens surface is required. The technology illustrated in FIG. 2Acan be used for numerous purposes including: (i) the determination ofthe presence of a rear facing child seat 230, (ii) the monitoring of therear of an occupant's head 242, (iii) the monitoring of the position ofoccupant 240, (iv) the monitoring of the position of the occupant'sknees 241, (v) the monitoring of the occupant's position relative to theairbag 250, (vi) the measurement of the occupant's height, as well asother monitoring functions as described elsewhere herein.

FIG. 2B is a perspective view corresponding to the embodiment shown inFIG. 2A illustrating some of the transducer mounting locations(including sensor 110A). The passenger side airbag module is designated104A and is mounted in the dashboard or instrument panel of the vehicle.

The occupant position sensor in any of its various forms is integratedinto the airbag system circuitry as shown schematically in FIG. 3. Inthis example, the occupant position sensors are sued as an input to asmart electronic sensor and diagnostic system. The electronic sensordetermines whether one or more of the airbags should be deployed basedon the vehicle acceleration crash pulse, or crush zone mounted crashsensors, or a combination thereof, and the occupant position sensordetermines whether the occupant is too close to any of the airbags andtherefore that the deployment should not take place. In FIG. 3, theelectronic crash sensor located within the sensor and diagnostic unitdetermines whether the crash is of such severity as to requiredeployment of one or more of the airbags. The occupant position sensorsdetermine the location of the vehicle occupants relative to the airbagsand provide this information to the sensor and diagnostic unit that thendetermines whether it is safe to deploy each airbag and/or whether thedeployment parameters should be adjusted. The arming sensor, if one ispresent, also determines whether there is a vehicle crash occurring. Insuch a case, if the sensor and diagnostic unit and the arming sensorboth determine that the vehicle is undergoing a crash requiring one ormore airbags and the position sensors determine that the occupants aresafely away from the airbag(s), the airbag(s), or inflatable restraintsystem, is deployed.

A particular implementation of an occupant position sensor having arange of from 0 to 2 meters (corresponding to an occupant position offrom 0 to 1 meter since the signal must travel both to and from theoccupant) using infrared is illustrated in the block diagram schematicof FIG. 4. The operation is as follows. A 48 MHz signal, f1, isgenerated by a crystal oscillator 401 and fed into a frequency tripler402 which produces an output signal at 144 MHz. The 144 MHz signal isthen fed into an infrared diode driver 403 which drives the infrareddiode 404 causing it to emit infrared light modulated at 144 MHz and areference phase angle of zero degrees. The infrared diode 404 isdirected at the vehicle occupant. A second signal f2 having a frequencyof A48.05 Mhz, which is slightly greater than f1, is similarly fed froma crystal oscillator 405 into a frequency tripler 406 to create afrequency of 144.15 MHz. This signal is then fed into a mixer 407 whichcombines it with the 144 MHz signal from frequency tripler 402. Thecombined signal from the mixer 407 is then fed to filter 408 whichremoves all signals except for the difference, or beat frequency,between 3 times f1 and 3 times f2, of 150 kHz. The infrared signal whichis reflected from the occupant is received by receiver 409 and fed intopre-amplifier 411, a resistor 410 to bias being coupled to theconnection between the receiver 409 and the pre-amplifier 411. Thissignal has the same modulation frequency, 144 MHz, as the transmittedsignal but now is out of phase with the transmitted signal by an angle xdue to the path that the signal took from the transmitter to theoccupant and back to the receiver. The output from pre-amplifier 411 isfed to a second mixer 412 along with the 144.15 MHz signal from thefrequency tripler 406. The output from mixer 412 is then amplified by anautomatic gain amplifier 413 and fed into filter 414. The filter 414eliminates all frequencies except for the 150 kHz frequency, however,now has a phase angle x relative to the signal from filter 408. Both 150kHz signals are now fed into a phase detector 415 which determines themagnitude of the phase angle x. It can be shown mathematically that,with the above values, the distance from the transmitting diode to theoccupant is x/345.6 where x is measured in degrees and the distance inmeters. The velocity can also be obtained using the distance measurementas represented by 416. An alternate method of obtaining distanceinformation, as discussed above, is to use the teachings of the McEwanpatents discussed above.

The applications described herein have been illustrated using the driverof the vehicle. The same systems of determining the position of theoccupant relative to the airbag apply to front and rear seatedpassengers, sometimes requiring minor modifications. It is likely thatthe sensor required triggering time based on the position of theoccupant will be different of the driver than for the passenger. Currentsystems are based primarily on the driver with the result that theprobability of injury to the passenger is necessarily increased eitherby deploying the airbag too late or by failing to deploy the airbag whenthe position of the driver would not warrant it but the passenger'sposition would. With the use of occupant position sensors for both thepassenger and driver, the airbag system can be individually optimizedfor each occupant and result in further significant injury reduction. Inparticular, either the driver or passenger system can be disabled ifeither the driver or passenger is out-of-position or if the passengerseat is unoccupied.

There is almost always a driver present in vehicles that are involved inaccidents where an airbag is needed. Only about 30% of these vehicles,however, have a passenger. If the passenger is not present, there isusually no need to deploy the passenger side airbag. The occupantmonitoring system, when used for the passenger side with proper patternrecognition circuitry, can also ascertain whether or not the seat isoccupied, and if not, can disable the deployment of the passenger sideairbag and thereby save the cost of its replacement. The same strategyapplies also for monitoring the rear seat of the vehicle. Also, atrainable pattern recognition system, as used herein, can distinguishbetween an occupant and a bag of groceries, for example. Finally, therehas been much written about the out-of-position child who is standing orotherwise positioned adjacent to the airbag, perhaps due to pre-crashbraking. The occupant position sensor described herein can prevent thedeployment of the airbag in this situation as well as in the situationof a rear facing child seat as described above.

The use of trainable pattern recognition technologies such as neuralnetworks is an important part of the instant invention, although othernon-trained pattern recognition systems such as fuzzy logic,correlation, Kalman filters, and sensor fusion (a derivative of fuzzylogic) can also be used. These technologies are implemented usingcomputer programs to analyze the patterns of examples to determine thedifferences between different categories of objects. These computerprograms are derived using a set of representative data collected ruingthe training phase, called the training set. After training, thecomputer programs output a computer algorithm containing the rulespermitting classification of the objects of interest based on the dataobtained after installation in the vehicle. These rules, in the form ofan algorithm, are implemented in the system that is mounted onto thevehicle. The determination of these rules is central to the patternrecognition techniques used in this invention. Artificial neuralnetworks using back propagation are thus far the most successful to therule determination approaches, however, research is underway to developsystems with many of the advantages of back propagation neural networks,such as learning by training, without the disadvantages, such as theinability to understand the network and the possibility of notconverging to the best solution . In particular, back propagation neuralnetworks will frequently give an unreasonable response when presentedwith data than is not within the training data. It is well known thatneural networks are good at interpolation but poor at extrapolation. Acombined neural network fuzzy logic system, on the other hand, cansubstantially solve this problem. Additionally, there are many otherneural network systems in addition to back propagation. In fact, onetype of neural network may be optimum for identifying the contents ofthe passenger compartment and another for determining the location ofthe object dynamically.

In some implementation of this invention, such as the determination thatthere is an object in the path of a closing window as described below,the rules are sufficiently obvious that a trained researcher can look atthe returned optical signals and device an algorithm it make therequired determinations. In others, such as the determination of thepresence of a rear facing child seat or an occupant, artificial neuralnetworks are frequently used to determine the rules. One such set ofneural network software for determining the pattern recognition rules,is available from the NeuralWare Corporation of Pittsburgh, Pa. Numerousbooks and articles, including more that 500 U.S. patents, describeneural networks in great detail and thus the theory and application ofthis technology is well known and will not be repeated here. Except in afew isolated situations where neural networks have been used to solveparticular problems limited to engine control, for example, they havenot heretofore been applied to automobiles and trucks.

The system generally used in the instant invention, therefore, for thedetermination of the presence of a rear facing child seat, an occupant,or an empty seat is the artificial neural network or a neural-fuzzysystem. In this case, the network operates on the returned signals formthe CCD array as sensed by transducers 110, 111, 113 and 114 (not shown)in FIG. 5, for example. For the case of the front passenger seat, forexample, through a training session, the system is taught todifferentiate between the three cases. This is done by conducting alarge number of experiments where available child seats are placed innumerous positions and orientations on the front passenger seat of thevehicle. Similarly, a sufficiently large number of experiments are runwith human occupants and with boxes, bags of groceries and otherobjects. AS many as 1,000,000 such experiments are run before the neuralnetwork is sufficiently trained so that it can differentiate among thethree cases and output the correct decision with a very highprobability.

Once the network is determined, it is possible to examine the result todetermine, form the algorithm created by the NeuralWare software, therules that were finally arrived at by the trial and error trainingtechnique. In that case, the rules can then be programmed into amicroprocessor. Alternately, a neural computer can be used to implementthe net directly. In either case, the implementation can be carried outby those skilled in the art of pattern recognition using neuralnetworks. If a microprocessor is used, a memory device is also requiredto store data from the analog to digital converters which digitize thedata from the receiving transducers. On the other hand, if a neuralnetwork computer is used, the analog signal can be fed directly from thetransducers to the neural network input nodes and an intermediate memoryis not required. Memory of some type is needed to store the computerprograms in the case of the microprocessor system and if the neuralcomputer is used for more than one task, a memory is needed to store thenetwork specific values associated with each task.

There are several methods measuring the height of the driver for use inautomatically adjusting the seat or for adjusting the seatbelt anchoragepoint. Some alternatives are shown in FIG. 5, which is a side plan viewof the front portion of the passenger compartment showing three heightmeasuring transducers or sensors 110, 111, 113, all of which are mountedon or near the headliner. These transducers may already be presentbecause of other implementations of the vehicle interior identificationand monitoring system described herein. The combination of fourtransducers can determine, by the methods described above, the locationof the head with great accuracy.

Optical transducers suing CCD arrays are now becoming price competitiveand, as mentioned above, will soon be the technology of choice forinterior vehicle monitoring. A single CCD array of 160 by 160 pixels,for example, coupled with the appropriate trained pattern recognitionsoftware, can be used to form an image of the head of an occupant andaccurately locate the head for some of the purposes of this invention.

The position of the rear of the head can also be known once the locus ofthe head has been determined. This information can be sued to determinethe distance from the headrest to the rearmost position of theoccupant's head and to control the position of the headrest so that itis properly positioned behind the occupant's head to offer optimumsupport in the event of a rear impact. Although the headrest of mostvehicles is adjustable, it is rare for an occupant to position itproperly, if at all. Each year, there are in excess of 400,000 whiplashinjuries in vehicles impacts approximately 90,000 of which are from rearimpacts (source: National Highway Traffic Safety Administration,(NHTSA)). A properly positioned headrest could substantially reduce thefrequency of such injuries that can be accomplished by the head detectorof this invention. The head detector is connected to the headrestcontrol mechanism and circuitry 540. This mechanism is capable of movingthe headrest up and down and, in some cases, rotating it fore and aft.Thus, the control circuitry 120 may be coupled to headrest controlmechanism and circuitry 540 to adjust the headrest based on thedetermined location of the rear of the occupant's head.

An occupant position sensor for side impacts used with a door mountedairbag system is illustrated at 530 in FIG. 5. This sensor has theparticular task of monitoring the space adjacent to the door-mountedairbag. Sensor 530 may also be coupled to control circuitry 120 whichcan process and use for information provided by sensor 530 in thedetermination of the location or identity of the occupant or location ofa part of the occupant.

Seatbelts are most effective when the upper attachment point to thevehicle is positioned vertically close to the shoulder of the occupantbeing restrained. If the attachment point is too low, the occupantexperiences discomfort from the rubbing of the belt on his or hershoulder. If it is too high, the occupant may experience discomfort dueto the rubbing of the belt against his or her neck and the occupant willmove forward by a greater amount during a crash which may result in hisor her head striking the steering wheel. For these reasons, it isdesirable to have the upper seatbelt attachment point located slightlyabove the occupant's shoulder. To accomplish this for various sizedoccupants, the location of the occupant's shoulder must be known, whichcan be accomplished by the vehicle interior monitoring system describedherein.

Such a system is illustrated in FIG. 6, which is a side planer view of aseatbelt anchorage adjustment system. In this system, infraredtransmitter and CCD array receivers 620 and 621 are positioned in aconvenient location proximate the occupant's shoulder, such as inconnection with the headliner, above and usually to the outside of theoccupant's shoulder. An appropriate pattern recognition system, as maybe resident in control circuitry 120 to which the receivers 620,621 arecoupled, as described above is then used to determine the location andposition of the shoulder. This information is provided by controlcircuitry 120 to the seatbelt anchorage height adjustment system 632(through a conventional coupling arrangement), shown schematically,which moves the attachment point 631 of the seatbelt 630 to the optimumvertical location for the proper placement of the seatbelt 630.

FIG. 7 is an angular perspective overhead view of a vehicle 710 about tobe impacted in the side by an approaching vehicle 720, where vehicle 710is equipped with an anticipatory sensor system showing a transmitter 730transmitting electromagnetic, such as infrared, waves toward vehicle720. This is one example of many of the use of the instant invention forexterior monitoring. The transmitter 730 is connected to an electronicmodule 740. Module 740 contains circuitry 742 to drive transmitter 730and circuitry 744 to process the returned signals from receivers 734 and736 which are also coupled to module 740. Circuitry 744 contains aprocessor such as a neural computer 745, which performs the patternrecognition determination based on signals from receivers 734 and 736(FIG. 7A). Receivers 734 and 736 are mounted onto the B-Pillar of thevehicle and are covered with a protective transparent cover. Analternate mounting location is shown as 738 which is in the door windowtrim panel where the rear view mirror (not shown) is frequentlyattached. One additional advantage of this system is the ability ofinfrared to penetrate fog and snow better than visible light which makesthis technology particularly applicable for blind spot detection andanticipatory sensing applications. Although it is well known thatinfrared can be significantly attenuated by both fog and snow, it isless so than visual light depending on the frequency chosen. (See forexample L. A. Klein, Millimeter-Wave and Infrared Multisensor Design andSignal Processing, Artech House, Inc, Boston 1997, ISBN 0-89006-764-3which is included herein by reference). I

The same system can also be used for the detection of objects in theblind spot of the vehicle and the image displayed for the operator tosee, or a warning system activated, if the operator attempts to changelanes for example. In this case, the mounting location must be chosen toprovide a good view along the side of the vehicle in order to pick upvehicles which are about to pass vehicle 710. Each of the locations 734,736 and 730 provide sufficient field of view for this applicationalthough the space immediately adjacent to the vehicle could be missed.Alternate locations include mounting onto the outside rear view mirroror the addition of a unit in the rear window or C-Pillar. The mirrorlocation, however, does leave the device vulnerable to being coveredwith ice, snow and dirt.

In both cases of the anticipatory sensor and blind spot detector, theinfrared transmitter and CCD array system provides mainly imageinformation to permit recognition of the object in the vicinity ofvehicle 710. To complete the process, distance information is alsorequire as well as velocity information, which can in general beobtained by differentiating the position data. This can be accomplishedby any one of the several methods discussed above, such as with a pulsedlaser radar system, as well as with a radar system.

Radar systems, which may not be acceptable for use in the interior ofthe vehicle, are now commonly used in sensing applications exterior tothe vehicle, police radar being one well-known example. Miniature radarsystems are now available which are inexpensive and fit within theavailable space. Such systems are disclosed in the McEwan patentsdescribed above. Another advantage of radar in this application is hatit is easy to get a transmitter with a desirable divergence angle sothat the device does not have to be aimed. One particularly advantageousmode of practicing the invention for these cases, therefore, is to useradar and a second advantageous made is the pulsed laser radar system,along with a CCD array, although the use of two CCD arrays or theacoustical systems are also good choices. The acoustical system has thedisadvantage of being slower than the laser radar device and must bemounted outside of the vehicle where it may be affected by theaccumulation of deposits onto the active surface.

In a preferred implementation, transmitter 730 is an infraredtransmitter and receivers 734, 736 and 738 are CCD transducers thatreceive the reflected infrared waves from vehicle 720. In theimplementation shown in FIG. 7, an exterior airbag 790 is shown whichdeploys in the event that a side impact is about to occur as describedin copending U.S. patent application Ser. No. 08/247,760cross-referenced above.

FIG. 8 illustrates the exterior monitoring system for use in detectingthe headlights of an oncoming vehicle or the taillights of a vehicle infront of vehicle 810. In this embodiment, the CCD array is designed tobe sensitive to visible light and a separate source of illumination isnot used. Once again for some applications, the key to this technologyis the use of trained pattern recognition algorithms and particularlythe artificial neural network. Here, as in the other cases above and inthe co-pending patent applications referenced above, the patternrecognition system is trained to recognize the pattern of the headlightsof an oncoming vehicle or the tail lights of a vehicle in front ofvehicle 810 and to then dim the headlights when either of theseconditions is sensed. It is also trained to not dim the lights for otherreflections such as reflections off of a sign post or the roadway. Oneproblem is to differentiate taillights where dimming is desired fromdistant headlights where dimming is not desired. Three techniques areused: (i) measurement of the spacing of the light sources, (ii)determination of the location of the light sources relative to thevehicle, and (iii) use of a red filter where the brightness of the lightsource through the filter is compared with the brightness of theunfiltered light. In the case of the taillight, the brightness of thered filtered and unfiltered light is nearly the same while there is asignificant difference for the headlight case. In this situation, eithertwo CCD arrays are used, one with a filter, or a filter which can beremoved either electrically, such as with a liquid crystal, ormechanically.

The headlights of oncoming vehicles frequently make it difficult for thedriver of a vehicle to see the road and safely operate the vehicle. Thisis a significant cause of accidents and much discomfort. The problem isespecially severe during bad weather where rain can cause multiplereflections. Visors are now used to partially solve this problem butthey do so by completely blocking the view through a large portion ofthe window and therefore cannot be used to cover the entire windshield.Similar problems happen when the sur is setting or rising and the driveris operating the vehicle in the direction of the sun. The vehicleinterior monitoring system of this invention can contribute to thesolution of this problem by determining the position of the driver'seyes as discussed above. If separate sensors are used to sense thedirection of the light from the on-coming vehicle or the sun and throughthe use of electro-chromic glass or a liquid crystal assembly, a portionof the windshield can be darkened to impose a filter between the eyes ofthe driver and the light source. Electrochromic glass is a materialwhere the color of the glass can be changed through the application ofan electric current. By dividing the windshield into a controlled gridor matrix of contiguous areas and through feeding the current into thewindshield for orthogonal directions, selective portions of thewindshield can be darkened as desired using either the electrochromic,liquid crystal or a similar technology. There are other technologiescurrently under development that perform in a similar manner as liquidcrystals. The term “liquid crystal” as used herein, therefore, will beused to represent the class of all such materials where the opticaltransmissibility can be varied electrically or electronically.Electrochromic products are available from Gentex of Zeeland, Mich. andDonnelly of Holland, Mich.

FIG. 9 illustrates how such a system operates. A sensor 910 located onvehicle 902 determines the direction of the light from the headlights ofoncoming vehicle 904. Sensor 910 is comprised of a lens and a CCD arraywith appropriate electronic circuitry which determines which elements ofthe CCD array are being most brightly illuminated. An algorithm storedin control circuitry 120 then calculates the direction of the light fromthe oncoming headlights based on the information from the CCD array.Transducers 110, 111, 113 and 114 determine the probable location of theeyes of the operator 101 of vehicle 902 in a manner such as describedabove. In this case, however, the determination of the probable locus ofthe driver's eyes is made with an accuracy of a diameter for each eye ofabout 3 inches (7.5 cm). This calculation sometimes will be in error andprovision is made for the driver to make an adjustment to correct forthis error as described below.

The windshield 916 of vehicle 902 comprises a liquid crystal, or similartechnology, and is selectively darkened at area 918 due to theapplication of a current along perpendicular directions 922 and 924 ofwindshield 916 (See FIG. 9A). The particular portion of the windshieldto be darkened is determined by control circuitry 120. Once thedirection of the light form the oncoming vehicle is known and thelocations of the driver's eyes are known, it is a matter of simpletrigonometry to determine which areas of the windshield matrix should bedarkened to impose a filter between the headlights and the driver's.This is accomplished by control circuitry 120. A separate controlsystem, not shown, located on the instrument panel, or at some otherconvenient location, allows the driver to select the amount of darkeningaccomplished by the system from no darkening to maximum darkening. Inthis manner, the driver can select the amount of light which is filteredto suit his particular physiology. The sensor 910 can either be designedto respond to a single light source or to multiple light sources to besensed and thus multiple portions of the vehicle windshield to bedarkened.

As mentioned above, the calculations of the location of the driver'seyes may be in error and therefore provision can be made to correct forthis error. One such system permits the driver to adjust the center ofthe darkened portion of the windshield to correct for such errorsthrough a knob on the instrument panel, steering wheel or otherconvenient location. Another solution permits the driver to make theadjustment by slightly moving his head. Once a calculation as to thelocation of the driver's eyes has been made, that calculation is notchanged even through the driver moves his head slightly. I is assumedthat the driver will only move his head to center the darkened portionof the windshield to optimally filter the light from the oncomingvehicle. The monitoring system will detect this initial head motion andmake the correction automatically for future calculations.

In the applications discussed and illustrated above, the source andreceiver of the electromagnetic radiation have been mounted in the samepackage. This is not necessary and in some implementations, theillumination source will be mounted elsewhere. For example, a laser beamcan be used which is directed along an axis which bisects the angelbetween the center of the seat volume and two of the arrays. Such a beammay come from the A-Pillar, for example. The beam, which may besupplemental to the main illumination system, provides a pointreflection from the occupying item that, in most cases, can been seen bytwo receivers. Triangulation thereafter can precisely determination thelocation of the illuminated point. This point can be moved to providemore information. In another case where it is desired to track the headof the occupant, for example, several such beams can be directed at theoccupant's head during pre-crash braking or even during a crash toprovide the fastest information as to the location of the head of theoccupant for the fastest tracking of the motion of the occupant's head.Since only a few pixels are involved, even the calculation time isminimized.

In most of the applications above the assumption has been made thateither a uniform field of light or a scanning spot of light will beprovided. This need not be the case. The light that is emitted ortransmitted to illuminate the object can be structured light. Structuredlight can take many forms starting with, for example, a rectangular orother macroscopic pattern of light and dark can be superimposed on thelight by passing it through a filter. If a similar pattern is interposedbetween the reflections and the camera, a sort of pseudo-interferencepattern can result sometimes known as Moire patterns. A similar effectcan be achieved by polarizing transmitted light so that different partsof the object that is being illuminated are illuminated with light ofdifferent polarization. Once again by viewing the reflections through asimilarly polarized array, information can be obtained as to where thesource of light came from which is illuminating a particular object.Thus, any of the transmitter/receiver assemblies or transducers in anyof the embodiments above using optics can be designed to use structuredlight.

One consideration when using structured light is that the source ofstructured light cannot be exactly co-located with the array because inthis case, the pattern projected will not change as a function of thedistance between the array and the object and thus the distance betweenthe array and the object cannot be determined. Thus, it is usuallynecessary to provide a displacement between the array and the lightsource. For example, the light source can surround the array, be on topof the array or on one side of the array. The light source can also havea different virtual source, i.e., it can appear to come from behind ofthe array or in front of the array.

The goal is to determine the direction that a particular ray of lighthad when it was transmitted from the source. Then by knowing whichpixels were illuminated by the reflected light ray along with thegeometry of the vehicle, the distance to the point of reflection off ofthe object can be determined. This is particularly effective if thelight source is not collated with the CCD array. If a particular lightray, for example, illuminates an object surface which is near to thesource then the reflection off of that surface will illuminate a pixelat a particular point on the CCD array. If the reflection of the sameray however occurs from a more distant surface, then a different pixelwill be illuminated in the CCD array. In this manner the distance fromthe surface of the object to the CCD can be determined by triangulationformulas. Similarly if a given pixel is illuminated in the CCD from areflection of a particular ray of light from the transmitter, and if weknown the direction that that ray of light was sent from thetransmitter, then the distance to the object at the point of reflectingcan be determined. If each ray of light is individually recognizable andtherefore can be correlated to the angle at which it was transmitted,then a full three-dimensional image can be obtained of the object thatsimplifies the identification problem.

The coding of the light rays coming from the transmitter can beaccomplished in many ways. One method is to polarize the light bypassing the light through a filter whereby the polarization is acombination of the amount and angle of the polarization. This gives twodimensions that can therefore be used to fix the angle that the lightwas sent. Another method is to superimpose an analog or digital signalonto the light which could be done, for example, by using an addressablelight valve, such as a liquid crystal filter, electrochromic filter, or,preferably, a garnet crystal array. Each pixel in this array would becoded such that it could be identified at the CCD.

The technique described above is dependent upon either changing thepolarization or using the time domain to identify particulartransmission angles with particular reflections. Spatial patterns canalso be imposed on the transmitted light which generally goes under theheading of structured light. The concept is that if a pattern isidentifiable then either the direction of transmitted light can bedetermined or, if the transmission source is collated with the receiver,then the pattern expands as it travels toward the object and then, bydetermining the size of the received pattern, the distance to the objectcan be determined. In some cases Moier pattern techniques are utilized.

When the illumination source is not placed at the same location as thereceiving array, it is typically placed at an angle such as 45 degrees.At least two other techniques can be considered. One is to place theillumination source at 90 degrees to the CCD array. In this case onlythose surface elements that are closer to the receiving array thenprevious surfaces are illuminated. Thus significant information can beobtained as to the profile of the object. In fact, if no object isoccupying the seat, then there will be no reflections except from theseat itself. This provides a very powerful technique for determiningwhether the seat is occupied and where the initial surfaces of theoccupying item are located.

The particular radiation field of the transmitting transducer can alsobe important to some implementations of this invention. In sometechniques the object which is occupying the seat is the only part ofthe vehicle which is illuminated. Extreme care is exercised in shapingthe field of light such that this is true. For example, the objects areilluminated in such a way that reflections from the door panel do notoccur. Ideally if only the items which occupy the seat can beilluminated then the problem of separating the occupant from theinterior vehicle passenger compartment surfaces can be more easilyaccomplished.

Another variant on the invention is to use no illumination source atall. In this case, the entire visible and infrared spectrum will beused. CMOS arrays are now available with very good night visioncapabilities making it possible to see and image an occupant in very lowlight conditions.

A further consideration to this invention is to use the motion of theoccupant, as determined from successive differential arrays, forexample, to help identify that there is in fact a living objectoccupying the seat, or for other purposes.

Thus, one method described above for determining the identification andposition of objects in a passenger compartment of a vehicle inaccordance wit the invention comprises the steps of transmittingelectromagnetic waves (optical or non-optical) into the passengercompartment from one or more locations, obtaining a plurality of imagesof the interior of the passenger compartment from several locations, andcomparing the images of the interior of the passenger compartment withsorted images representing different arrangements of objects in thepassenger compartment to determine which of the stored images match mostclosely to the images of the interior of the passenger compartment suchthat the identification of the objects and their position is obtainedbased on data associated with the stored images. The electromagneticwaves may be transmitted from transmitter/receiver assemblies positionedat different locations around a seat such that each assembly issaturated in a middle of a side of the ceiling surrounding the seat orin the middle of the headliner directly above the seat. The method wouldthus be operative to determine the identification and/or position of theoccupants of that seat. Each assembly may comprise an opticaltransmitter (such as an infrared LED, an infrared LED with a diverginglens, a laser with a diverging lens and a scanning laser assembly) andan optical array (such as a CCD array and a CMOS array). The opticalarray is thus arranged to obtain the images of the interior of thepassenger compartment represented by a matrix of pixels. To enhance themethod, prior to the comparison of the images, each obtained image oroutput from each array may be compared with a series of stored images orarrays representing different unoccupied states of the passengercompartment, such as different positions of the seat when unoccupied,and each stored image or array is subtracted from the obtained image oracquired array. Another way to determine which store image matches mostclosely to the images of the interior of the passenger compartment is toanalyze the total number of pixels of the image reduced below athreshold level, and analyze the minimum number of remaining detachedpixels. Preferably, a library of stored images is generated bypositioning an object on the seat, transmitting electromagnetic wavesinto the passenger compartment from one or more locations, obtainingimages of the interior of the passenger compartment, each from arespective location, associating the images with the identification andposition of the object, and repeating the positioning step, transmittingstep, image obtaining step and associating step for the same object indifferent positions and for different objects in different positions. Ifthe objects include a steering wheel, a seat and a headrest, the angleof the steering wheel, the telescoping position of the steering wheel,and angle of the back of the seat, the position of the headrest and theposition of the seat may be obtained by the image comparison. Oneadvantage of this implementation is that after the identification andposition of the objects are obtained, one or more systems in thevehicle, such as an occupant restraint device or system, a mirroradjustment system, a seat adjustment system, a steering wheel adjustmentsystem, a pedal adjustment system, a headrest positioning system, adirectional microphone, an air-condition/heating system, anentertainment system, may be affected based on the obtainedidentification and position of at least one of the objects. The imagecomparison may entail inputting the images or a form thereof into aneural network which provides for each image of the interior of thepassenger compartment, an index of a stored image that most closelymatches the image of the interior of the passenger compartment. Theindex is thus utilized to locate stored information from the matchedimage including inter alia, a locus of a point representative of theposition of the chest of the person, a locus of a point representativeof the position of the head of the person, one or both ears of theperson, one or both eyes of the person and the mouth of the person.Moreover, the position of the person relative to at least one airbag orother occupant restraint system of the vehicle may be determined so thatdeployment of the airbag(s) or occupant restraint system is controlledbased on the determined position of the person. It is also possible toobtain information about the location of the eyes of the person from theimage comparison and adjust the position of one or more of the rear viewmirrors based on the location of the eyes of the person. Also, thelocation of the eyes of the person may be obtained such that an externallight source may be filtered by darkening the windshield of the vehicleat selective locations based on the location of the eyes of the person.Further, the location so the ears of the person may be obtained suchthat a noise cancellation system in the vehicle is operated based on thelocation the ears of the person. The location of the mouth of the personmay be used to direct a directional microphone in the vehicle. Inaddition, the location of the locus of a point representative of theposition of the chest or head (e.g., the probable center of the chest orhead) over time may be monitored by the image comparison and one or moresystems in the vehicle controlled based on changes in the location ofthe locus of the center of the chest or head over time. This monitoringmay entail subtracting a most recently obtained image from animmediately preceding image and analyzing a leading edge of changes inthe images or deriving a correlation function which correlates theimages with the chest or head in an initial position with most recentlyobtained images. In one particularly advantageous embodiment, the weightapplied onto the seat is measured and one or more systems in the vehicleare affected (controlled) based on the measured weight applied onto theseat and the identification and position of the objects in the passengercompartment.

In another method disclosed above for determining the identification andposition of objects in a passenger compartment of a vehicle inaccordance with the invention, electromagnetic waves are transmittedinto the passenger compartment from one or more locations, a pluralityof images of the interior of the passenger compartment are obtained,each from a respective location, a three-dimensional map of the interiorof the passenger compartment is created from the images, and a patternrecognition technique is applied to the map in order to determine theidentification and position of the objects in the passenger compartment.The pattern recognition technique may be a neural network, fuzzy logicor an optical correlator or combinations thereof. The map may beobtained by utilizing a scanning laser radar system where the laser isoperated in a pulse mode and determining the distance from the objectbeing illuminated using range gating. (See for example, H. Kage, W.Freemen, Y Miyke, E. Funstsu, K. Tanaka, K. Kyuma “Artificial retinachips as on-chip image processors and gesture-oriented interfaces”,Optical Engineering, Dec., 1999, Vol. 38, Number 12, ISSN 0091-3286)

In a method disclosed above for tracking motion of a vehicularoccupant's head or chest in accordance with the invention,electromagnetic waves are transmitted toward the occupant from tat leastone location, a first image of the interior of the passenger compartmentis obtained from each location, the first image being represented by amatrix of pixels, and electromagnetic waves are transmitted toward theoccupant from the same location(s) at a subsequent time and anadditional image of the interior of the passenger compartment isobtained from each location, the additional image being represented by amatrix of pixels. The additional image is subtracted from the firstimage to determine which pixel shave changed in value. A leading edge ofeh changed pixels and a width of a filed of the changed pixels isdetermined to thereby determine movement of the occupant from the timebetween which the first and additional images were taken. The firstimage is replaced by the additional image and the steps of obtaining anadditional image and subtracting the additional image from the firstimage are repeated such that progressive motion of the occupant isattained.

A method disclosed above for controlling deployment of an occupantrestraint system in a vehicle comprise the steps of transmittingelectromagnetic waves toward an occupant seated in a passengercompartment of the vehicle from one or more locations, obtaining aplurality of images of the interior of the passenger compartment, eachfrom a respective location, analyzing the images to determine thedistance between the occupant and the occupant restraint system, andcontrolling deployment of the occupant restraint system based on thedetermined distance between the occupant and the occupant restraintsystem. The images may be analyzed by comparing the images of theinterior of the passenger compartment with stored images representingdifferent arrangements of objects in the passenger compartment todetermine which of the stored images match most closely to the images ofthe interior of the passenger compartment, each stored image havingassociated data relating to the distance between the occupant in theimage and the occupant restraint system. The image comparison step mayentail inputting the images or a form thereof into a neural networkwhich provides for each image of the interior of the passengercompartment, an index of a stored image that most closely matches theimage of the interior of the passenger compartment. In a particularlyadvantageous embodiment, the weight of the occupant on a seat ismeasured and deployment of the occupant restraint system is controlledbased on the determined distance between the occupant and the occupantrestraint system and the measured weight of the occupant.

In another method disclosed above for determining the identification andposition of objects in a passenger compartment of a vehicle, a pluralityof images of he interior of the passenger compartment, each from arespective location and of radiation emanating from the objects in thepassenger compartment, and the images of the radiation emanating fromthe objects in the passenger compartment are compared with stored imagesof radiation emanating from different arrangements of objects in thepassenger compartment to determine which of the stored images match mostclosely to the images of the interior of the passenger compartment suchthat the identification of the objects and their position is obtainedbased on data associated with the stored images. In this embodiment,there is no illumination of the passenger compartment withelectromagnetic waves. Nevertheless, the same processes described abovemay be applied in conjunction with this method, e.g., affecting anothersystem based on the position and identification of the objects, alibrary of stored images generated, external light source filtering,noise filtering, occupant restraint system deployment control and theutilization of weight for occupant restraint system control.

Thus, disclosed above is a system to identify, locate and monitoroccupants, including their parts, and other objects in the passengercompartment and objects outside of a motor vehicle, such as anautomobile or truck, by illuminating the contents of the vehicle and/orobjects outside of the vehicle with electromagnetic radiation, andspecifically infrared radiation, or using radiation naturally emanatingfrom the object, and using one or more lenses to focus images of thecontents onto one or more arrays of charge coupled devices (CCD's) orCMOS arrays. Outputs from the CCD or CMOS arrays are analyzed byappropriate computational means employing trained pattern recognitiontechnologies, to classify, identify or locate the contents and/orexternal objects. In general, the information obtained by theidentification and monitoring system may be used to affect the operationof at least one other system in the vehicle.

When the vehicle interior monitoring system in accordance with someembodiments of this invention is installed in the passenger compartmentof an automotive vehicle equipped with a passenger protective device,such as an inflatable airbag, and the vehicle is subjected to a crash ofsufficient severity that the crash sensor has determined that theprotective device is to be deployed, the system determines the positionother vehicle occupant relative to the airbag and disables deployment ofthe airbag if the occupant is positioned so that he/she is likely to beinjured by the deployment of the airbag. In the alternative, theparameters of the deployment of the airbag can be tailored to theposition of the occupant relative to the airbag, e.g., a depowereddeployment.

In some implementations of the invention, several CCD or CMOS arrays areplaced in such a manner that the distance from, and the motion of theoccupant toward, the airbag can be monitored as a transverse motionacross the field of the array. In this manner, the need to measure thedistance from the array to the object is obviated. In otherimplementations, the source of infrared light is a pulse modulated laserwhich permits an accurate measurement of the distance to the point ofreflection through the technique of range gating to measure the time offlight of the radiation pulse.

In some applications, a trained pattern recognition system, such as aneural network or neural-fuzzy system, is used to identify the occupancyof the vehicle or an object exterior to the vehicle. In some of thesecases, the pattern recognition system determines which of a library ofimages most closely matches the seated state of a particular vehicleseat and thereby the location of certain parts of an occupant can beaccurately estimated form the matched images, thus removing therequirement for the pattern recognition system to locate the head of anoccupant, for example.

There has thus been shown and described, among other things, amonitoring system for monitoring both the interior and the exterior ofthe vehicle using an optical system with one or more CCD arrays andother associated equipment which fulfills all the objects and advantagessought after. Many changes, modifications, variations and other uses andapplications of the subject invention will, however, become apparent tothose skilled in the art after considering this specification and theaccompanying drawings which disclose the preferred embodiments thereof.All such changes, modifications, variations and other uses andapplications which do not depart from the spirit and scope of theinvention are deemed to be covered by the invention which is limitedonly by the following claims.

We claim:
 1. A vehicle including an arrangement for determining vehicleoccupant position relative to a fixed structure within the vehicle, thearrangement comprising an array structured and arranged to receive animage of a portion of the passenger compartment of the vehicle in whichthe occupant is likely to be situated, a lens arranged between saidarray and the portion of the passenger compartment, adjustment means forchanging the image received by said array, and processor means coupledto said array and said adjustment means for determining upon changing bysaid adjustment means of the image received by said array when the imageis clearest whereby a distance between the occupant and the fixedstructure is obtainable based on the determination by said processormeans when the image is clearest.
 2. The vehicle of claim 1, whereinsaid adjustment means are arranged to adjust said lens to thereby changethe image received by said array.
 3. The vehicle of claim 2, whereinsaid lens has a variable focal length, said adjustment means beingarranged to adjust the focal length of said lens.
 4. The vehicle ofclaim 1, wherein said array is arranged on an A-pillar of the vehicle.5. The vehicle of claim 1, wherein said array is arranged at a positionabove a top surface of an instrument panel of the vehicle.
 6. Thevehicle of claim 1, wherein said array is arranged on an instrumentpanel of the vehicle and oriented to receive an image reflected by awindshield of the vehicle.
 7. The vehicle of claim 1, wherein said arrayis a CCD array.
 8. The vehicle of claim 7, further comprising a liquidcrystal or electrochromic glass filter coupled to said array forfiltering the image of the portion of the passenger compartment.
 9. Thevehicle of claim 1, wherein said array is a CMOS array.
 10. The vehicleof claim 1, further comprising an occupant protection device coupled tosaid processor means, said processor means being arranged to controlsaid occupant protection device based on the distance between theoccupant and the fixed structure.
 11. The vehicle of claim 10, whereinsaid occupant protection device is an airbag whereby deployment of saidairbag is controlled by said processor means.
 12. A vehicle including anarrangement for determining at least one of vehicle occupant presence,type and position relative to a fixed structure within the vehicle, thevehicle having a front seat and an A-pillar, the arrangement comprisinga first array mounted on the A-pillar of the vehicle and arranged toreceive an image of a portion of the passenger compartment in which theoccupant is likely to be situated, and processor means coupled to saidfirst array for determining the at least one of vehicle occupantpresence, type and position based on the image of the portion of thepassenger compartment received by said first array.
 13. The vehicle ofclaim 12, wherein said processor means are arranged to utilize a patternrecognition technique.
 14. The vehicle of claim 12, further comprising asecond array arranged to receive an image of at least a part of the sameportion of the passenger compartment as said first array, said processormeans being coupled to said second array and arranged to determine theat least one of vehicle occupant presence, type and position based onthe images of the portion of the passenger compartment received by saidfirst and second arrays.
 15. The vehicle of claim 14, wherein saidsecond array is arranged at a central portion of a headliner of thevehicle between sides of the vehicle.
 16. The vehicle of claim 12,further comprising a reactive component, system or subsystem coupled tosaid processor means, said processor means being arrange to control saidreactive component, system or subsystem based on the determination ofthe at least one of vehicle occupant presence, type and position. 17.The vehicle of claim 16, wherein said reactive component, system orsubsystem is an airbag assembly including at least one airbag, saidprocessor means being arranged to control at least one deploymentparameter of said at least one airbag.
 18. The vehicle of claim 12,wherein said first array is a CCD array.
 19. The vehicle of claim 18,further comprising a liquid crystal or electrochromic glass filtercoupled to said first array for filtering the image of the portion ofthe passenger compartment.
 20. The vehicle of claim 12, wherein saidfirst array is a CMOS dynamic pixel camera.
 21. The vehicle of claim 12,wherein said first array is an active pixel camera.
 22. The vehicle ofclaim 12, wherein said first array is an HDRC camera.
 23. The vehicle ofclaim 12, wherein said processor means determining the vehicle occupantpresence, type and position based on the image of the portion of thepassenger compartment received by said first array.
 24. The vehicle ofclaim 12, wherein said processor means comprise a trained neural networktrained in a training phase by obtaining data of images of the portionof the passenger compartment with different occupants in associationwith information about the presence, type and position of the occupantssuch that said trained neural network is applied using as input theimage of the portion of the passenger compartment during operation toobtain the information about at least one of the presence, type andposition of the occupant.
 25. The vehicle of claim 12, furthercomprising a transmitter for transmitting light into the portion of thepassenger compartment.
 26. The vehicle of claim 12, wherein the light isin the infrared portion of the electromagnetic spectrum.
 27. A vehicleincluding an arrangement for obtaining information about a vehicleoccupant within the vehicle, the arrangement comprising a lighttransmitter structured and arranged to transmit structured light into aportion of the passenger compartment in which the occupant is likely tobe situated, an array arranged to receive an image of the portion of thepassenger compartment, and processor means coupled to said array foranalyzing the image of the portion of the passenger compartment toobtain information about the occupant.
 28. The vehicle of claim 27,wherein said transmitter and said array are spaced from and proximateone another and the information obtained about the occupant is adistance from the location of said transmitter and said array.
 29. Thevehicle of claim 27, wherein said processor means are arranged toutilize a pattern recognition technique.
 30. The vehicle of claim 27,further comprising a reactive component, system or subsystem coupled tosaid processor means, said processor means being arranged to controlsaid reactive component, system or subsystem based on the informationobtained about the occupant.
 31. The vehicle of claim 30, wherein saidreactive component, system or subsystem is an airbag assembly includingat least one airbag, said processor means being arranged to control atleast one deployment parameter of said at least one airbag.
 32. Thevehicle of claim 27, wherein said array is a CCD array.
 33. The vehicleof claim 27, wherein said processor means comprise a trained neuralnetwork trained in a training phase by obtaining data of images of theportion of the passenger compartment with different occupants inassociation with information about the occupants such that said trainedneural network is applied using as input the image of the portion of thepassenger compartment during operation to obtain the information aboutthe occupant.
 34. The vehicle of claim 27, wherein said structured lightis polarized light such that different parts of the portion of thepassenger compartment in which the occupant is likely to be situated areilluminated with light of different polarization.
 35. A vehicleincluding an arrangement for determining vehicle occupant positionrelative to a fixed structure within the vehicle, the arrangementcomprising an array structured and arranged to receive an image of aportion of the passenger compartment of the vehicle in which theoccupant is likely to be situated, a lens, adjustment means foradjusting the focal length of said lens relative to said array, andprocessor means coupled to said array and said adjustment means fordetermining upon adjustment of the focal length relative to said arrayby said adjustment means when the image is clearest whereby a distancebetween the occupant and the fixed structure can be obtained based onthe determination by said processor means when the image is clearest.36. A vehicle including an arrangement for determining a position of anexterior object relative to the vehicle, the arrangement comprising anarray structured and arranged to receive an image of an exteriorenvironment surrounding the vehicle containing at least one object, alens arranged between said array and the exterior environment,adjustment means for changing the image received by said array, andprocessor means coupled to said array and said adjustment means fordetermining upon changing by said adjustment means of the image receivedby said array when the image is clearest whereby a distance between theobject and the vehicle is obtainable based on the determination by saidprocessor means when the image is clearest.
 37. The vehicle of claim 36,wherein said adjustment means are arranged to adjust said lens tothereby change the image received by said array.
 38. The vehicle ofclaim 37, wherein said lens has a variable focal length, said adjustmentmeans being arranged to adjust the focal length of said lens.
 39. Thevehicle of claim 36, wherein said array is a CCD array.
 40. The vehicleof claim 39, further comprising a liquid crystal or electrochromic glassfilter coupled to said array for filtering the image of the exteriorenvironment.
 41. The vehicle of claim 3, wherein said array is a CMOSarray.
 42. The vehicle of claim 36, further comprising an occupantprotection device coupled to said processor means, said processor meansbeing arranged to control said occupant protection device based on thedistance between the object and the vehicle.
 43. The vehicle of claim42, wherein said occupant protection device is an airbag wherebydeployment of said airbag is controlled by said processor means.
 44. Thevehicle of claim 36, wherein said array is mounted on the vehicle toreceive an image of a blind spot of a driver of the vehicle.
 45. Avehicle including an arrangement for determining a position of anexterior object relative to the vehicle, the arrangement comprising anarray structured and arranged to receive an image of an exteriorenvironment surrounding the vehicle containing at least one object, alens, adjustment means for adjusting the focal length of said lensrelative to said array, and processor means coupled to said array andsaid adjustment lens for determining upon adjustment of the focal lengthof said lens by said adjustment means when the image is clearest wherebya distance between the object and the vehicle is obtainable based on thedetermination by said processor means when the image is clearest.
 46. Avehicle including an arrangement for determining vehicle occupantposition relative to a fixed structure within the vehicle, thearrangement consisting of: a single array structured and arranged toreceive an image of a portion of the passenger compartment of thevehicle in which the occupant is likely to be situated, a single lensset arranged between said array and the portion of the passengercompartment, adjustment means for changing the image received by saidarray, and processor means coupled to said array and said adjustmentmeans for determining upon changing by said adjustment means of theimage received by said array when the image is clearest whereby adistance between the occupant and the fixed structure is obtainablebased on the determination by said processor means when the image isclearest.
 47. A vehicle including an arrangement for determining aposition of an exterior object relative to the vehicle, the arrangementconsisting of: a single array structured and arranged to receive animage of an exterior environment surrounding the vehicle containing atleast one object, a single lens set arranged between said array and theexterior environment, adjustment means for changing the image receivedby said array, and processor means coupled to said array and saidadjustment means for determining upon changing by said adjustment meansof the image received by said array when the image is clearest whereby adistance between the object and the vehicle is obtainable based on thedetermination by said processor means when the image is clearest.